Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge Networks
Authors: Bowen Jin
Yu Zhang
Yu Meng
Jiawei Han
Venue: ICLR 2023 poster (2023)

Abstract: Edges in many real-world social/information networks are associated with rich text information (e.g., user-user communications or user-product reviews). However, mainstream network representation learning models focus on propagating and aggregating node attributes, lacking specific designs to utilize text semantics on edges. While there exist edge-aware graph neural networks, they directly initialize edge attributes as a feature vector, which cannot fully capture the contextualized text semantics of edges. In this paper, we propose Edgeformers, a framework built upon graph-enhanced Transformers, to perform edge and node representation learning by modeling texts on edges in a contextualized way. Specifically, in edge representation learning, we inject network information into each Transformer layer when encoding edge texts; in node representation learning, we aggregate edge representations through an attention mechanism within each node’s ego-graph. On five public datasets from three different domains, Edgeformers consistently outperform state-of-the-art baselines in edge classification and link prediction, demonstrating the efficacy in learning edge and node representations, respectively.

Published as a conference paper at ICLR 2023
EDGEFORMERS: GRAPH-EMPOWERED TRANSFORM- ERS FOR REPRESENTATION LEARNING ON TEXTUAL-
EDGE NETWORKS
Bowen Jin, Yu Zhang, Yu Meng, Jiawei Han Department of Computer Science, University of Illinois at Urbana-Champaign (bowenj4, yuz9, yumeng5, hanj) ) @ Willinoiis .dd
ABSTRACT
Edges in many real-world social/information networks associated with rich text information (e.g., user-user communications or user-product reviews). However, mainstream network representation learning models focus on propagating and aggregating node attributes, lacking specific designs to utilize text semantics on edges. While there exist edge-aware graph neural networks, they directly initialize edge attributes as feature vector, which cannot fully capture the contextualized text semantics of edges. In this paper, we propose Edgeformen a framework built upon graph-enhanced Transformers, to perform edge and node representation learning by modeling texts on edges in a contextualized way. Specifically, in edge representation learning, inject network information into each Transformer layer when encoding edge texts; in node representation learning, we aggregate edge representations through attention mechanism within each node's ego-graph. On five public datasets from three different domains, Edgeformers consistently outperform state-of-the-an baselines in edge classification and link prediction, demonstrating the efficacy in learning edge and node representations, respectively.
INTRODUCTION
Networks are ubiquitous and are widely used to model interrelated data in the real world, such as user-user and user-item interactions on social media (Kwak et al.) 2010 Leskovec et al. 2010) and recommender systems (Wang et al. 2019 Jin et al. 2020p. In recent years, graph neural networks (GNNs) (Kipf & Welling 2017 Hamilton et al. 2017] Velickovic et al. 2018 Xu et al. 2019) have demonstrated their power network representation learning. However, vast majority of GNN models leverage node attributes only and lack specific designs to capture information refer to these models as node-centric GNNs.) Yet, in many scenarios, there rich information associated with edges in network. For example, when person replies to another social media, there will be directed edge between them accompanied by the response texts; when a user comments on an item, the user's review will be naturally associated with the user-item edge.
edges. (We
To utilize edge information during network representation learning, some edge-aware GNNs (Gong & Cheng 2019 Tiang et al. 2019) Yang & Li, 2020 Jo et al. 2021) have been proposed. Nevertheless, these studies assume the information carried by edges can be directly described as an attribute vector. This assumption holds well when edge features are categorical (e.g., bond features in molecular graphs (Hu et al. 2020) and relation features in knowledge graphs (Schlichtkrull et al. 2018)). However, effectively modeling free-text edge information edge-aware GNNs has remained elusive, mainly because bag-of ofg-of-words and context-free embeddings (Mikolov et al. (013)) used in previous edge-aware GNNs cannot fully capture contextualized text semantics. For example, "Byzantine" in history book reviews and "Byzantine" in distributed system papers should have different meanings given their context, but they correspond to the same entry in bag-of-words vector and have the same context-free embedding.
To accurately capture contextualized semantics, a straightforward idea is to integrate pretrained language models (PLMs) (Devlin et al. 2019: Liu al. 2019| Clark et al. 2020) with GNNs. In node-centric GNN studies, this idea has been instantiated by PLM-GNN cascaded architecture (Fang al. 2020 Li et al.
2021 Zhu al. 202I), 1, where text information is first encoded by PLM and then aggregated by a GNN. However, such architectures process text and graph signals one after
"Code found at http://tpp///.teeececc////ssssss

---

Published as a conference paper at ICLR 2023
the other, and fail to simultancously model the deep interactions between both types of information. This could be a loss to the text encoder because network signals are often strong indicators to text semantics. For example, brief political tweet may become more comprehensible if the stands of the two communicators are known. To deeply couple PL.Ms and GNNs, the recent GraphFormers model (Yang et al., 202T) proposes a GNN-nestee PI.M architecture inject network information into the text encoding process. They introduce GNNs nested in between Transformer layers so that the center node encoding not only leverages its own textual information, but also aggregates the signals from its neighbors. Nevertheless, they assume that only nodes are associated with textual information and cannot be easily adapted to handle text-rich edges.
To effectively model the textual and network structure information via a unified encoder architecture, in this paper, we propose a novel network representation learning framework, Edgeformers, that leverage graph-enhanced Transformers to model edge texts in a contextualized way. Edgeformers include two architectures, Edgeformer-E and Edgeformer-N, for edge and node representation learning, respectively. In Edgeformer-E, we add virtual node tokens to each Transformer layer inside the PLM when encoding edge texts. Such an architecture goes beyond the PLM-GNN cascaded architecture and enables deep, layer-wise interactions between network and text signals to produce edge representations. In Edgeformer-N, we aggregate the representations obtain node representations through attention mechanism within each node's ego-graph. The two architectures can be trained via edge classification (which relies on good edge representations) and link prediction (which relies on good node representations) tasks, respectively. To summarize, our main contributions are as follows:
edge
Conceptually, identify the importance of modeling text information on network edges and formulate the problem of representation learning on textual-edge networks.
Methodologically, we propose Edgeformers (i.e., Edgeformer-E and Edgeformer-N), two graph- enhanced Transformer architectures, to deeply couple network and text information in contextual- ized way for edge and node representation learning.
Empirically, we conduct experiments on five public datasets from different domains and demonstrate the superiority of Edgeformers over various baselines, including node-centric GNNs, edge-aware GNNs, and PLM-GNN cascaded architectures.
2
PRELIMINARIES
2.1 TEXTUAL- EDGE NETWORKS
In a textual-edge network, each edge is associated with texts. We view the texts on each edge as a document, and all such documents constitute corpus D. Since the major goal of this work is to explore the effect of textual information on edges, we assume there is no auxiliary information (e.g.. categorical or textual attributes) associated with nodes in the network.
Definition (Textual- Eeex Networks) A textual-edge network is defined as G = (V,E.D) where V. E, D represent the sets of nodes, edges, and documents, respectively. Each edge eij is associated with a document dij E 'D.
To give example of textual-edge networks, consider review network (e.g., Amazon (He & McAuleyy] 2016)) where nodes are users and items. If a user i writes a review about an item ej. there will be edge eij connecting them, and the review text will be the associated document dej.
2.2 TRANSFORMER
Many PLMs (e.g., BERT (Devlin et al. 2019g) adopt multi-layes Transformer architecture (Vaswani et al. 2017) to encode texts. Each Transformer layer utilizes a multi-head self-attention mechanism to
obtain a contextualized representation of each text token. Specifically, let H [ h .... h2!01
denote the output sequence of the (-th Transformer layer, where h (9) e Rd is the hidden representation of the text token at position i. Then, in the (7 + 1)-th Transformer layer, the multi-head self-attention (MHA) is calculated as
MHA(H ()) = head (")
(I)
==1
K Q!")
head' (H = v. softmax
(2)
Vd/k
Q."h = w 10, K!O- w K.
H (
vi =
(3)
2

---

Published as a conference paper at ICLR 2023
Node Reprosentation
Virtual Node Token Hidden States
Edge Representation
AGG
Virtual Local Network Hidden State
Local Network
Epo-Graph
(a) Edgeformer-E
(b) Edgeformer-N
Figure 1: Model Framework Overview. (a) An illustration of Edgeformer-E E for edge representation learning, where virtual node token hidden states are concatenated the edge text original token hidden states to inject network signal into edge text encoding. (b) An illustration of Edgeformer-N for node representation learning, where Edgeformer-E enhanced by local network structure virtual token hidden state and edge representations are aggregated to obtain node representation.
where Wa.:, WK. Wv., are query, key, and value matrices to be learned by the model, k is the number of attention head and is the concatenate operation.
2.3 PROBLEM DEFINITIONS
Our general goal is to learn meaningful edge and node embeddings in textual-edge networks as to benefit downstream tasks. To be specific, we consider the following two tasks focusing on edge representation learning and node representation learning, respectively.
The first task is edge classification, which relies on learning good representation he of an edge E. We assume each edge eij belongs category y y. The category can be indicated by its associated text dij and/or the nodes vy and ey- For example, in the Amazon review network, y (1-star, 2-star, .... 5-star). The category of eiy reflects how the user o is satisfied with the item Uy which may be expressed by the sentiment of dij and/or implied by U, preference and uy 's quality, Given review, the task is to predict its category based on review text and user/item information.
Definition 2 (Edge Classification) In textual-edge network G EV,E, D), we can observe the category of some edges Eirain E. Given an edge CiS E E Extraan predict its category y e y based on di E 'D and V4, U3 V.
The second task is link prediction, which relies on learning an accurate representation h of node E V. Given two nodes v; and ej. the task is to predict whether there is an edge between them. Note that, unlike edge classification, we no longer have the text information d,5 (because we even do not know whether Eij exists). Instead, we need to exploit other edges (local network structure) involving is or Uj as well as their text to learn node representations h., and hey. For example, the Amazon review network, we aim to predict whether user will be satisfied with product according to the user's 's reviews towards other products and the item's 's reviews from other users.
Definition 3
(Link Prediction) In textual-edge network G EV,E,D), Eirain and their associated text. Given Us, By E V where eij & Etrains predict whether eij E.
observe some edges
3 PROPOSED METHOD
In this section, present our Edgeformers framework. Based on the two tasks mentioned in Section 2.3 we first introduce how we conduct edge representation learning by jointly considering text and network information via Transformer-basee architecture (Edgeformer-E) Then, illustrate how to perform node representation learning using the edge representation learning module as building blocks (Edgeformer-N), The overview of Edgeformers. is shown in Figure

---

Published as a conference paper at ICLR 2023
3.1 EDGE REPRESENTATION LEARNING (EDGEFORMER -E)
Network-aware Edge Text Encoding with Virtual Node Tokens. Encoding dij in a textual-edge network is different from encoding plain text, mainly because edge texts are naturally accompanied by network structure information, which can provide auxiliary signals. Given that text semantics can be well captured by a multi-layer Transformer architecture (Devlin et al. 2019), we propose simple and effective way to inject network signals into the Transformer encoding process. The key idea is to introduce virtual node tokens. Given an edge Eij = (v,, v2) and its associated texts dij. let
He Rdx= denote the output representations of all text tokens in dij after the 1-th model layer (1 > 1). In each layer, we introduce two virtual node tokens to represent U; and vy, respectively. Their (1)
embeddings are denoted (!) zn; and 20j Rd. which are concatenated the text token sequence
hidden states as follows:
H (1) 20) 20
After the concatenation, Hey involving nodes i; and Vi. To let text token representations carry node signals, we adopt a multi-head attention mechanism:
(1) contains information from both eij's associated text dij and its
MHA(H
H
J r-1
head' (H
engle eegith:
(5)
Q (
K (1)
H E.
V
H Ey.e
(6)
In Eq. (5). the multi-head attention is asymmetric (i.e., the keys K and values V are augmented with virtual node embeddings but queries Q are not) to avoid network information being overwritten by text signals. This design has been used in existing studies Yang et al.) (202I). and offers better effectiveness than the original self-attention mechanism according to experiments in Section 4.2 The output of MHA includes updated node-aware representations of text tokens. Then, following the Transformer architecture (Vaswani et al.) (017). the updated representations will go through feed-forward network (FFN) to finish our (1 1)-th model layer encoding. Formally,
H Normalize H E.g Egg
MHAAH'
(7)
H (++1) Normalize
H O E.j +FFN(H Eq
(9))
(8)
where Normalize(.) is the layer normalization function. After L model layers, the final representation
of the [CLS] token will be used as the edge representation of eij, i.e., he,, = =He [CLS].
Representation of Virtual Node Tokens. The virtual node representation _!!!) used in Eq is
obtained by a layer-specific mapping of the initial node embedding (0) Formally,
( (0) 20s
where w!(!) Rdxx is the mapping matrix for the I-th layer. The large population of nodes will introduce a large number of parameters to our framework, which may finally lead to model underfitting. As result, in Edgeformers, we set the initial node embedding be low-dimensional (e.g., (0) ( and project it to the high- nigh-dimensional token representation space (e.g., z! e IR 768).
Note that it is possible to go beyond the linear mapping in Eq. (9) and use structure-aware encoders
such as GNNs to obtain 2ng and we leave such extensions for future studies.
3.2 TEXT-AWARE NODE REPRESENTATION LEARNING (EDGEFORMER-N)
In this section, we first discuss how to perform text-aware node representation learning by taking the aforementioned edge representation learning module (i.e., Edgeformer-E) building blocks. Then, we propose to enhance the edge representation learning module with the target node's additional local network structure.
Aggregating Edge Representations. Since the edge representations learned by Edgeformer-E capture both text semantics and network structure information, a straightforward way obtain node representation is to aggregate the representations of all edges involving the node. Given node V; its representation h. is given by
hv AGG(( ley Ne(e:)))
(10)
4

---

Published as a conference paper at ICLR 2023
where No. (et) is the set of edges containing v,- AGG(-) can be any permutation invariant function such as mean(() or max(-). Here, we instantiate AGG() with an attention- based aggregation:
Deg. H softmax Wozeo)) h
Σ
Gegp Begs
(11)
where W, Rdxd' is a learnable scoring matrix. Enhancing Edge Representations with the Node's Local Network Structure. Since we are aggregating information from multiple edges, it is intuitive that they can mutually improve each other's representation by providing auxiliary semantic signals. For example, given a conversation about "Transformers" and their participants' other conversations centered around "machine learning", is more likely that the term "Transformers" refers to deep learning architecture rather than a character in the movie. To implement this intuition in the edge representation learning module,
introduce the third virtual token hidden state
(1())
ezg during edge encoding:
z 20g esslus
(1) (1)
(1)
H (1)
(12)
is the contextualized representation of eij given target node v, 's local network structure. Now we introduce how to calculate h Herylo. by aggregating information from (N();)
where h "sslle
Representation of h (1) "eglo: For each edge eis e (v;) (including Ety). we treat the hidden state of
its [CLS] token after the 1-th layer as its representation (i.e., hes. H [CLS)). To obtain (1)
"4.loz"
adopt MHA let all edges in Ne(w;) interact with each other.
[.... h (1) Roll 1u
MHA ([.... her
(1) 1)
(13)
In the equation above, involving U.- Therefore, after MHA, the edge representation (000) from vy's " local network structure No (v.).
Regs
contains the 1-th layer representations of all edges
essentially aggregates information
cayl:;
Connection between Edgeformer-N and GNNs. According to Figure 1 Edgeformer-N adopts Transformer- based architecture. Meanwhile, it can also be viewed GNN model. Indeed, GNN models (Wu et al. 2020) Yang et al., 2020) mainly adopt a propagation-aggreeatin paradigm to obtain node representations:
((-1) PROP(1 (ni-"), h ( (Vj (N()): h.th AGG (n. (n.-"), " (())) (14) 1
Analogously, in Edgeformer-N, Eq. (13) can be treated as the propagation function PROP()) and the aggregation step (GG() is the combination of Eqs. 12). (8 and 10
3.3 TRAINING
we consider edge classification and link prediction two tasks train Edgeformer-E -E and Edgeformer-N, respectively.
As mentioned in Section 2.3
Edge Classification. For Edgeformer-E (i.e., edge representation learning), adopt supervised training, the objective function of which is as follows.
Le
Σ Yonr logüe,, + -ees)) Togg( &eey)) eis
(15)
where Deg [(heay) is the predicted category distribution of eij and f(-) is learnalle classifier. Link Prediction. For Edgeformer-N (i.e., node representation learning), we conduct unsupervised training, where the objective function is as follows.
exp(h, (hThhh.))))) exp(h hw) + EnT exp(h!.u.))
C, = Σ Σ -log DEV (v)
(16)
Here, (v) is the set of o's node neighbors and u' denotes random negative sample. In implementation, we utilize "in-bau ba-batcc negative samples" (Karpukhin et al., 2020) to reduce encoding and training costs.
5

---

Published as a conference paper at ICLR 2023
Table 1: Edge classification performance Amazon-Movie, Amazon-App. GhodreeassCrring and
Amazon-Movis
Amazon-Apps
Goodreads- Crime
Goodreads-Chiiee
Model
Macro-F1
Micro-FI
Macro-Fi Fl
Micro-FI
Macro-F1
Micro-F1
Macro-Fi
Micro-FI
TF-IDF 50.01
64.22
48.30
62.88
43.07
51.72
39.42
49.90
TF-IDF+nodes
53.59
66.34
50.56
65.08
49.35
57.50
47.32
56.78
BERT
61.38
71.36
59.11
69.27
56.41
61.29
51.57
57.72
BERT+nodes
63.00
72.45
59.72
70.82
58.64
65.02
54.42
60.46
Edgeformer-E
64.18
73.59
60.67
71.28
61.03
65.86
57.45
61.71
Overall Algorithm. The workflow of our edge representation learning (Edgeformer-E) and node representation learning (Edgeformer-N) algorithms can be found in Alg. T and Alg. 2 respectively.
Complexity Analysis. Given node involved in N edges, and each edge has P text tokens, the time complexity of edge encoding for each Edgeformer-E layer is ( (the same as one vanilla Transformer layer). The time complexity of node encoding for each Edgeformer-N layer is O(NP2 + N For most nodes in the network, we can assume N & N so the complexity is roughly ON P2) (the same as one PLM-GNN cascaded layer). For more discussions about time complexity, please refer to Section 4.5
EXPERIMENTS
In this section, we first introduce five datasets. Then, we demonstrate the effectiveness of Edgeformers on both edge-level (e.g., edge classification) and node-level (e.g., link prediction) tasks. Finally, we conduct visualization and efficiency analysis to further understand Edgeformers.
4.1 DATASETS
We run experiments on three worl-world networks: Amazon (He & McAuley 2016), Goodreads (Wan al. 2019), and StackOverflou Amazon user-item interaction network, where reviews are treated as text edges; Goodreads is reader-book network, where readers' comments are used as edge text information; StackOverflow is an expert-question network, and there will an edge when an expert posts an answer question. Since Amazon and Goodreads both have multiple domains, we select two domains for each of them. In total, there are five datasets used in evaluation (i.e., Amazon-Movia Amazon-Apps, Goodreads-Crime statistics can be found in appendix A.I
Goodreads-Childrre StackOverflow), Dataset
4.2 TASK FOR EDGE REPRESENTATION LEARNING
Baselines. We compare our Edgeformer-E model with a oag-of-wordde method (TF-IDF (Robertson) & Walker 1994)) and a pretrained language model (BERT (Devlin et al.] (009)). Both baselines are further enhanced with network information by concatenating the node embedding 2; with the bag-of-words vector (TF-IDF+nodes) or appending it to the input token sequence (BERT+nodes).
Edge Classification. The model asked to predict the category of each edge based on its associated text and local network structure. There are 5 categories for edges in Amazon (i.e., 1-star, .... 5-star) and 6 categories for edges in Goodreads (i.e., O-star, 5-star).
For TF-IDF methods, the dimension of the bag-of-words vector is 2000. BERT-involved models and Edgeformer-E have the same model size (L = 12, d = 768) and are initialized by the same checkpoinf" The dimension of initial node embeddings d' is set to be 64. We use AdamW as the optimizer with (c, 31, 32) = (1e-8, 0.9, 0.999). The learning rate 1e-5. The early stopping patience is epochs. The batch size is 25. Macro-F1 and Micro-F1 are used evaluation metrics. For BERT-involved models, parameters in BERT are trainable.
Table summarizes the performance comparison the five datasets. From the table, we observe that: (a) our Edgeformer-E consistently outperforms all the baseline methods; (b) PLM-based methods (i.e., BERT, node., and Edgeformer-E) can have more promising results than bag-of-words methods (i.e., TF-IDF and TF-IDF+nodes); (c) injecting node information can significantly improve performance if we compare TF-IDF with TF- or compare BERT with noRT+, (d) the performance of Edgeformer-E is better than that of directly appending node embeddings to the

---

Published as a conference paper at ICLR 2023
Table 2: Link prediction performance (on the testing set) on Amazon-Movie, Amazon-Apps, Goodreads- Goodreads- Chiilrrd--Chidrenn and StackOverflow. A denotes the relative improvement of our model comparing with the best baseline.
Amazon Movie
Amazon- AmazznnAAps
Goodreads- Crime
Goodreads Childree
StackOverflow
Model
MRR
NDCG
MRR
NDCG
MRR
NDCG
MRR
NDCG
MRR
NDCG
MF
0.2032
0.3546
0.1482
0.3052
0.1923
0.3443
0.1137
0.2716
0.1040
0.2642
MeunSAGE
0.2138
0.3657
0.1766
0.3343
0.1832
0.3368
0.1066
0.2647
0.1174
0.2768
MaxSAGE
0.2178
0.3694
0.1674
0.3258
0.1846
0.3387
0.1066
0.2647
0.1173
0.2769
GIN
0.2140
0.3648
0.1797
0.3362
0.1846
0.3374
0.1128
0.2700
0.1189
0.2778
CensNet
0,2048
0,3568
0.1894
0.3457
0.1880
0.3398
0.1157
0.2726
0.1235
0,2806
NENN
0,2565
0,4032
0.1996
0.3552
0,2173
0.3670
0.1297
0.2854
0.1257
0,2854
BERT
0,2391
0,3864
0.1790
0,3350
0.1986
0.3498
0.1274
0.2836
0,1666
0.3252
HERT+ MAXE
0,2780
0.4224
0.20855
01,6002
0.2193
0,3694
0,1312
0.2872
0.1681
0.3264
BERT+MeanSAGE
0,2491
0,3972
0.1983
0,3540
0.1952
0.3477
0.1223
0.2791
0.1678
0.3264
BERT+GIN
0,2573
0,4037
0,2000
0,3552
0.2007
0.3522
0.1238
0.2801
0.1708
0.3279
Graphforreers
0.2756
0.419%
0,2066
0,3607
0.2176
0,3684
0.1323
0.2887
0.1693
0.3278
BERT+CensNet
0.1919
0.3462
0.1544
0,3132
0.1437
0.3000
0.0847
0,2436
0.1173
0,2789
BERT+NEEN
0.2821
0,4256
0.2127
0.3666
0.2262
0.3756
0. 1365
0.2925
0.1619
0.3215
Edgeformer-N
0,2919
0,4344
0.2239
0.3771
0,2395
0,3875
0.1446
0,3000
0,1754
0_3339
+ F %6
3.5%
2.1%%
5.3%
2.9%
55.9%
3.2%
5.9%
2.6%
2.7%
1.88%
input token sequence (i.e. BERT+nodes), possibly because network information overwritten by text signals in BERT+nodes' deeper layers.
4.3 TASKS FOR NODE REPRESENTATION LEARNING
Baselines. We compare Edgeformer-N N with several vanilla GNN models and PLM-integrated GNN models. Vanilla GNN models include node-centric GNNs such as MeanSAGE. (Hamilton et al. 2017), MaxSAGE (Hamilton al. 2017) and GIN (Xu et al. 2019), and edge-aware GNNs such as CensNet (Jiang et al. 2019) and NENN (Yang & Li 2020). All vanilla edge-aware GNNs models use bag-of-words as initial edge feature representations. PLM-integrated GNN models utilize a PLM (Devlin et al. 2019) to obtain text representations on edges and adopt a GNN to obtain node representations by aggregating edge representations. Baselines include BERT+MeanS AGE (Hamilton et al.) 2017). BERT+MaXSAGE. (Hamilton et al., 2017). BERT+GIN (Xu et al.. 2019). BERT+CensNer (Jiang et al. 2019), BERT+NENN YYag & Li 2020), and GraphFormers (Yang) et al. 2021). To verify the importance of both text and network information in text-rich networks, we also include matrix factorization (MF) (Qiu et al. 2018) and vanilla BERT (Devlin et al., 2019) in the comparison.
Link Prediction. The task is to predict whether there will be an edge between two target nodes, given their local network structures. Specifically, in the Amazon and Goodreads datasets, given the target user's reviews to other items/books and the target item/book's ss reviews from other users, we aim to predict whether there will be a 5-star link between the target user and the target item/book. In the SterkOverllow dataset, we aim to predict whether the target expert can give an answer to the target question. We use MRR and NDCG as evaluation metrics.
For vanilla GNN models, we find that adopting MF node embeddings as initial node embeddings can help them obtain better performance (Lv et al. 202T). For edge-aware GNNs, bag-of-words vectors are used as edge features, the size of which is set as 2000. For BERT-involved models, the training parameters are the same as 4.2 During the testing stage, all methods are evaluated with samples in the batch for efficiency, iee., each query node is provided with one positive key node and 99 randomly sampled negative key nodes. More details can be found in Appendix A.8
Table 2 shows the performance comparison. From the table, we can find that: (a) Edgeformer-N out- performs all the baseline methods consistently; (b) BERT-based methods can have significantly better performance than bag-of-words GNN methods, which demonstrates the importance of contextualized text semantics encoding; (c) edge-awan aware methods can have better performance, but it depends on how the edge information contributes to node representation learning; (d) our Edgeformer-N is the best since it takes edge text into consideration and deeply integrates text encoding and local network structure encoding.
Ablation Study. We further conduct an ablation study to validate the effectiveness of all the three virtual tokens on node representation learning. The three virtual token hidden states are deleted respectively from the whole model and the results are shown in T From the table, we can find that Edgeformer-N generally outperforms all the model variants all the datasets, except for that
7

---

Published as a conference paper at ICLR 2023
Table 3: Ablation study of link prediction performance (on the testing set) Amazon-Movie Amazon-Apps. Goodreads-Cring, Gooilreds-Chiidren, and StackOverflow. (-) means removing the
corresponding virtual tokens.
Amazon Movie
Amazon-Appi
Goodreads Crime
Goodreads Children
StackOverflow
Model
MRR
NDCG
MRR
NDCG
MRR
NDCG
MRR
NDCG
MRR
NDCG
Edgefoormr-NN
0.2919
0.4344
0.2239
0.3771
0,2395
0,3875
0.1446
0.3000
0.1754
0.3339
center node token
0,2899
0,4325
0,2178
0.3717
0.2361
0,3847
0,1407
0.2964
0.1702
0.3291
-neighhor node token
0.2880
0.4306
0.2115
0.3656
0.2322
0.3807
0.1411
0.2963
0.1730
0.3310
-neighbor text token
0.2895
0.4321
0.2260
0.3789
0.2386
0.3867
0.1442
0.2998
0.1734
0.3314
Table 4: Node classification performance on Amazon-Movie and Amazon- App.
Amazon- Movie
Amazon-Apps
Model
Macro-Fl
Micro-F1
PREC
Macro-F1
Micro-FI
PREC
MF
0.7566e0.000 0017
0.8234=00.013
0,4647+0.0151 000041
0.8393+0.0012
0.8462+0.0008
CensNet
0.8528.0.0010
0.883910.000 0.089
0.884510.000) 0007
0.278240.0168
0.827910.0006
0.833110.0005
NENN
0.918620.0008
0.9341=0.000
0.9347#0.0007
0,3408
0.8789=0.0019
0.8819=0.0017
BERT
0.9209#0.0005
0.936740.0003 0.0061+0.0003
0.760810.0175
0,92K340.0015
0.9337=0.0015
BERT+CensNet
0.9032=000006
0.9221=0.0004
00227+0.0004
0,5750:0.0277
0.8692=0.00334
0,8731=0.002
BERT-NENT
0,9247=0.0005
0.9387=0.0004
0.9393+0.0000 0005
0.7556+0.000 0.092
0,9306+0.0000 0.000
0.9382+0.000 0006
I:dgeformer-N
0.927620.0007
0.9417=0.0005 0.9411=0.0006
0.7758+0.0100
0,9339=0.0007
0.9431=0.0005
without neighbor edge information virtual token in Amazon-Apps, which indicates the importance of all three virtual token hidden states.
Node Classification with unsupervised node embedding. To further evaluate the quality of the unsupervised learned node embeddings, we fix the node embeddings obtained from link prediction and train a logistic regression classifier to predict nodes' categories. This is a multi-class multi-label classification task, where there are 2 classes for Amazon-Movie and 26 classes for Amazon-Apps, Table 4 summarizes the performance comparison between several edge-aware methods. We can find that: (a) Edgeformer-N can outperform all the baselines significantly and consistently, which indicates that Edgeformer-N -N can learn more effective node representations; (b) edge-aware models can have better performance, but it depends on how the edge text information is employed.
4.4 EMBEDDING VISUALIZATION
To reveal the relation between edge embeddings and node embeddings learned by our model, we apply t-SNE (Van der Maaten & Hinton 2008) to visualize them in Figure Node embeddings (i.e., (hole e V)) are denoted as stars, while edge embeddings (i.e,, {hele E E are denoted as points with the same color as the node they link to. From the figure, we observe that: (I) node embeddings tend to be closer to each other in the embedding space compared with edge embeddings; (2) the embeddings of edges linked to the same node are in the vicinity of each other.
4.5 EFFICIENCY ANALYSIS
We now compare the efficiency of BERT+GIN (a node-centric GNN), BERT+NENN (an edge-aware GNN), GraphFormers (a PLM-GNN nested architecture), and our Edgeformer-N. All models are run on one NVIDIA A6000. The mini-batch size is 25; each sample contains one center node and [w) neighbor edges; the maximum text length is 64 tokens. The running time (per matc-batch) of compared models is reported in Table 5 where we have the following findings: (a) the time cost of training Edgeformer-N is quite close to that of BERT+GIN, BERT+NENN, and GraphFormers; (b) PLM-GNN nested architectures (i.e., GraphFormers and Edgeformer-N) require slightly longer time during training than PLM-GNN cascaded architectures (i.e., BERT+GIN and BERT+NENN); (c) the time cost of Edgeformer-N increases linearly with the neighbor size [N (v)), J, which is consistent with our analysis in Section 3.3 that the time complexity of Edgeformer-N is (NN + N2) ~ O(N p2)
when N <<< P.
5 RELATED WORK
5.1 PRETRAINED LANGUAGE MODELS
PLMs are proposed to learn universal language representations from large-scale text corpora. Early studies such as word2vec Mikooov et al. 2013), fastText (Bojanowski et al. 2017). and GloVe (Pennington et al. 2014) aim to learn set of context-inder indexxt-nneeenndent word embeddings to capture word semantics. However, many NLP tasks are beyond werd-level, so it is beneficial to derive word representations based specific contexts. Contextualized language models are extensively studied
8

---

Published as a conference paper at ICLR 2023
-22
(a) Apps
(b) Children
Figure 2: Embedding visualization. Node embeddings are denoted as stars, and the embeddings of edges are denoted as points with the same color if they are linked to the same node.
Table 5: Time cost (ms) per mini-batch for BERT+GIN, BERT+NENT GraphFormers, and Edgeformer-N, with neighbor size (w) increasing from 2 to 5 on Amazon-Apps. Goodreads- Children, and SterkOverllow. Edgeformer-N achieves similar efficiency with the baselines.
Amazon-Appp
Goodreads Children
StackOverllow
Model
2
5
4
3
BERT-GIN
15.53
21.44
25.69
31.19
15.44
21.38
25.88
31.02
15.29
21.41
25.58
31,05
BERT-NENN
15.70
21.71
26.03
31.50
15.78
21.86
15
31.46
15.74
21.97
26.09
31.46
GraphFormers
17.21
23.56
34.43
17.08
23.76
28.43
34.38
17.13
23.65
Edgeformer-N
18.68
25.39
30.45
36.57
18.74
25.17
30.42
36.42
18.57
25.32
30.46
36.32
recently to achieve this goal. For example, GPT (Peters et al.
2018
Radford et al., 2019) adopts
auto-regressive language modeling to predict token given all previous tokens; BERT (Devlin et al.)
2019) and RoBERTa (Liu et al. 2019) are trained via masked language modeling to recover randomly masked tokens; XLNet (Yang et al. 2019) proposes permutation language modeling; ELECTRA (Clark et al. 2020) uses an auxiliary Transformer to replace some tokens and pretrains the main Transformer detect the replaced tokens. For more related studies, can refer to recent survey (Qiu et al. 2020). To jointly leverage text and graph information, previous studies (Zhang et al., 2019 al. 2020 Li al. 2021 Zhu et al. (02T) propose PLM-GNN cascaded architecture, where the text information of each node is first encoded via PL, then the node representations are aggregated via GNNs. (Bi et al. 2022) proposes a triple2seq operation to lincarize subgraphs and a "mask prediction" paradigm to conduct inference. Recently, GraphFormers (Yang et al. 202T) introduces GNN-nested Transformer to stack GNN layers and Transformer layers alternately. However, these works mainly consider textual-node networks, thus their focus is orthogonal to ours on textual-edge networks.
5.2 EDGE-AWARE GRAPH NEURAL NETWORKS
A vast majority of GNN models (Kipf & Welling 2017 Hamilton et al. 2017 Velickovic et al. 2018 Xu et al. 2019) leverage node attributes only and lack specific designs utilize edge features. Heterogeneous GNNs (Schlichtkrull et al., 2018 Yang et al. 2020 assume each edge has a pre- defined type and take such types into consideration during aggregation. However, they still cannot deal with more complicated features (e.g., text) associated with the edges. EGNN (Gong & Cheng 2019) introduces an attention mechanism inject edge features into node representations: CensNet (Jiang et 2019) alternately updates node embeddings and edge embeddings in convolution layers; NENN (Yang& & Li 2020) aggregates the representation of each node/edge from both its node and edge neighbors via a GAT-like attention mechanism. EHGNN (Jo et al. (02T) proposes the dual hypergraph transformation and conducts graph convolutions for edges. Nevertheless, these models do not collaborate PLMs and GNNs to specifically deal with text features edges, thus they underperform our Edgeformers model, even stacked with a BERT encoder.
CONCLUSIONS
We tackle the problem of representation learning on textual-edge networks. To this end, we propose a novel graph-empowered Transformer framework, which integrates local network structure information into each Transformer layer text encoding for edge representation learning and aggregates edge representation fused by network and text signals for node representation. Comprehensive experiments on five real-world datasets from different domains demonstrate the effectiveness of Edgeformers on both edge-level and node-level tasks. Interesting future directions include (1) exploring other variants of introducing network signals into Transformer text encoding and (2) applying the framework to more network-related tasks such recommendation and text-rich social network analysis.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank anonymous reviewers for their valuable and insightful feedback. Research was sup- ported in part by US DARPA KAIROS Program No. PA8750-19-2-100 1004 and INCAS Program No. HR001 121C0165, National Science Foundation I18-19-56151, IIS-17-41317, and IIS 17-04532, and the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Enviromment (I-GUIDE) by NSF under Award No. 2118329. Any opinions, findings, and conclusions or recommendations expressed herein are those the authors and do not necessarily represent the views, either expressed implied, of DARPA or the U.S. Government.
REFERENCES
Abubakar Abid, Maheen Farooqi, and James Zou, Persistent anti-muslim bias in large language models. In AIES, pp. 298-306, 2021.
Zhen Bi, Siyuan Cheng, Ningyu Zhang, Xiaozhuan Liang, Feiyu Xiong, and Huajun Chen. Relphormer: Relational graph transformer for knowledge graph representation. arXiv preprint arXiv:2205.108852 2022.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. TACL, 5:135 146, 2017.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning, Electra: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-IILT, pp. 4171-4186, 2019.
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. Hierarchical graph network for multi- hop question answering. In EMNLP, pp. 8823 8823-8838, 2020.
Liyu Gong and Qiang Cheng. Exploiting edge features for graph neural networks. In CVPR, pp. 9211-92199 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pp. 1024-1034, 2017.
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In WWW, pp. 507-517, 2016.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurlPS, pp. 12-2233,,,, 2020.
Xiaodong Jiang, Pengsheng Ji, and Sheng Li. Censnet: Convolution with edge-node switching in graph neural networks. IJCAI, pp. 2656-2662, 2019.
Bowen Jin, Chen Gao, Xiangnan He, Depeng Jin, and Yong Li. Multi-behavior recommendation with graph convolutional networks. In SIGIR, pp. 659-668, 2020.
Jachycong Jo, Jinheon Back, Seul Lee, Dongki Kim, Minki Kang, and Sung Ju Hwang. Edge representation learning with hypergraphs. In NeurIPS, pp. 7534-7546, 2021.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint S1t::2000.049006 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
10

---

Published as a conference paper at ICLR 2023
Haewoon Kwak, Changhyun Lee, Hosung Park, and Sue Moon. What is twitter, social network or news media? In WWW, pp. 591-600, 2010.
Jure Leskovec, Daniel Huttenlocher, and Jon Kleinberg. Predicting positive and negative links in online social networks. In www, pp. 641-650, 2010.
Chaozhuo Li, Bochen Pang, Yuming Liu, Hao Sun, Zheng Liu, Xing Xie, Tiaqqi Yang, Yanling Cui, Liangjie Zhang, and Qi Zhang. Adsgnn: Behavior-graph augmented relevance modeling in sponsored search. In SIGIR, pp. 223-232, 2021.
Paul Pu Liang, Chiyu Wu, Louis-Phillipe Morency, and Ruslan Salakhutdinov. Towards understand- ing and mitigating social biases in language models. In ICML, pp.
2021.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 1907v 1692, 2019.
Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng. Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang. Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In KDD, pp. 1150-1160, 2021.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. NIPS, pp. 3111-3119, 2013.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543, 2014.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, pp. 2227-2237, 2018.
Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In WSDM, pp. 459-469-4 460,-66,,200 2018.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10): 1872-1897, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Stephen E Robertson and Steve Walker. Some simple effective approximations the 2-poisson model for probabilistic weighted retrieval. In SIGIR, pp. 232-241, 1994.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In ESWC, pp. 607, 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-6008, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lið, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
Mengting Wan, Rishabh Misra, Ndapandula Nakashole, and Julian McAuley, Fine-grained spoiler detection from seage-scale review corpora. In ACL, pp. 2605-26100 2019.
Xiang Wang. Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In SIGIR, pp. 165-174, 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengyi Zhang, and Philip S Yu. A comprehensive survey graph neural networks. IEEE TKDE, 12(1):4-24, 2020.
11

---

Published as a conference paper at ICLR 2023
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? ICLR, 2019.
Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. Heterogeneous network representa- tion learning: A unified framework with survey and benchmark. IEEE TKDE, 2020.
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. In NeurlPS, 2021.
Yulci Yang and Dongsheng Li. Nenn: Incorporate node and edge features in graph neural networks. In ACML, pp. 593 508,608, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, In NeurIPS, pp. 5754-5764, 2019.
Chuxu Zhang, Ananthram Swami, and Nitesh V Chawla. Shne: Representation learning for semantic- associated heterogeneous networks. In WSDM, pp. 690-698, 2019.
Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang, Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. Textgnn: Improving text encoder via graph neural network in sponsored search. In WWW, pp. 2848 2888-285, 2021.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASETS
The statistics of the five datasets can be found in Table 6
Table Dataset Statistics
Dataset
Node
# Edge
Amazon-Moon
173,986
1,697,533
A Amazon-Apps
100,468
752,937
Goodreads-Crima
385,203
1,849,236
Goodreads-Childree
192,036
734,640
StackOverflow 129,322
281,657
A.2 SUMMARY OF EDGEFORMERS' ENCODING PROCEDURE
Algorithm 1: Edge Representation Learning Procedure of Edgeformer-E
Input
TTee edge exj. its associated text d.y. and its involved nodes and vy. The initial token embeddings Hes of the document d.y.
Output: embedding fees of the edge eij-
begin
obtain 1ayer-1 representation on of text takens (0)
H! Normalize H
MHA (0) (0)) )
(1) - Normalize H
+ FFN
)):
// obtain the base embedding of i and b
(0) < Embeddingge::);
z", < Eunbedding(r;);
for 1,... do
// ostain layer-l representation of virtual node tckens
20 _!))
2e,
20 (0)
1/ optain layer (+1) zepresentation of text tokens
H (
H
!! Normalize H + MHA ( H
H (++1) Normalize H. + FFN (HFN"HH!)) (H
(1+1)
(0'
end
return bay
!!!+1)
(L+1) (((+))(CSS)) [CLS]:
end
A.3 EDGE CLASSIFICATION
We also compare our method with the state-of-the-art edge representation learning method EHGNN (Jo et al. 2021) and two node-centric PLM-GNN methods BERT+MaXSAGE (Hamilton et al. 2017) and GraphFormers (Yang al. 202T). The experimental results can be found in Table 7 From the results, we can find that Edgeformer-E consistently outperforms all the baseline methods, including EHGNN, BERT+EHGNN, BERT+MaxSABB and Fraphrormers.
EHGNN cannot obtain promising results because of two reasons: 1) edge-edge propagation: EHGNN proposes to transform edge to node and node to edge in the original network, followed by graph convolutions on the new hypernetwork. This results in edge-edge information propagation when conducting edge representation learning. However, edge-edge information propagation has the underlying edge-edge homophily assumption which is not always true in textual-edge networks. For example, when predicting the rate review text e,j given by u to i, it is not straightforward to make the judgment based on reviews for i written by other (neighbor edges); 2) The integration
13

---

Published as a conference paper at ICLR 2023
Algorithm 2: Node Representation Learning Procedure Edgeformer-N.
Input
:hee center node v., its edge neighbors Ne(v:), and its node neighbors N..(s)).
The initial token embedding Has of each document de associated with e (c.)).
Output: embedding h., of the center node v;-
begin
obtain layer-1 representation of text tokens for csy E (N(())) do
H !0) < Normalize H
MHA (0) H ( ) FFN (H ));
H (1)
Hes Normalize H
end
// obtain the base embedding of each node
for E (1:) {e.)
(0) (mbedding(u));
Zu
end
for
L.... Lo
// obtain layer-1 representation virtual node tckens
for E (w) u (0:
-0) ! end
// obtain layer-I representation of virtual neighber aggregation tckens
for Gs E N (w; do (
H Hej [CLS];
end
h
Paylm ... calo - MHA (
!/ obtain layer-(I+1) representation of text tokens
for es E (w; do
H
H
H!l Normalize H MHA ( H
H (++)) Normalize (9'
+ FFN ( ));
end
end
// obtain the edge representation
for E Ne(w:) do
heyle - H (1++1)( end
// obtain the node representation h = AGG((heyllylee N.(w:)))
return hv:
end
14

---

Published as a conference paper at ICLR 2023
of text and network signals are loose for BERT+EHGNN, since such architectures process text and graph signals one after the other, and fail to simultaneously model the deep interactions between both types of information. However, our Edgeformer-E E is designed following the more reasonable node-edge homophily hypothesis and deeply integrating text network signals by introducing virtual node tokens in Transformer encoding.
Note that both PLM+GNN and Edgeformer-E require textual information on ALL nodes in the network. However, this assumption does hold in many textual-edge networks. Therefore, propose a way around to concatenate the text on the edges linked to the given node together to make up node text. However, such a strategy does lead competitive performance of PLM++MxSAAE and GraphFormers according to our experimental results. Therefore, to make our model generalizable the case of missing node text, the proposed Edgeformers can be a better solution.
Table 7: Edge classification performance on Amazon-Movie, Amazon-App. Goodreads-Crinne and Goodreads- Chhnldden......
Amazon-Movie
Amazon-Apor
Goodreads-Crine Crime
Goodreads-Chine
Model
Macro-Fl
Micro-F1
Macro-F1
Micro-F1
Macro-F1
Micro-Fl
Macro-Fl
Micro-Fl
TF-IDF
50.01
64.22
48.30
62.88
43.07
51.72
39.42
49.90
TF-IDF+nodes
53.59
66.34
50.56
65.08
49.35
57.50
47.32
56.78
EHGNN
49.90
64.04
48.20
63.63
44.49
52.30
40.01
50.25
BERT
61.38
71.36
$9.11
69.27
56.41
61.29
51.57
57.72
BERT+noder
63.00
72.45
$9.72
70.82
58.64
65.02
54.42
60.46
BERT+EHGNN
61.45
70.73
70.79
$6.92
61.66
$2.46
$7.97
MAXGAE
61.57
70.79
58.95
70.45
57.20
61.98
52.75
58.53
GraphFormers
61.73
71.52
$9.67
70.19
57.49
62.37
52.93
58.34
Edgeformer-E
64.18
73.59
60.67
71.28
61.03
65.86
57.45
61.71
A.4 LINK PREDICTION
We further report the link prediction performance of compared models on the validation set in Table
Table 8: Link prediction performance (on the validation set) Amazon- Amazon-Apps. Goodreads- Goodreads- Children, and StackOverflow. A denotes the relative improvement of our model comparing with the best baseline.
Amazon-Mover
Amazon-Appa
Gondreads Crime
Choxdreads-hhldden
StackOverflow
Model
MRR
NDCG
MRR
NDCG
MRR
NDCG
MRR
NDCG
MRR
NDCG
MF
0,2178
0.3666
0.1523
0.3086
0.2492
0.3966
0.1470
0.3042
0.1104
0.2702
MeanSAGE
0.2280
0.3775
0.3375
0.2286
0.3792
0.1348
0.2927
0.2846
MaxSAGE
0,2321
0.3812
0.1708
0.3288
0.2299
0.3812
0,1339
0.2919
0.1257
0,2848
GIN
0.2287
0.3769
0.1846
0.3402
0.2306
0.3802
0.1420
0.2989
0.1275
0.2860
ConsNet
0,2186
0.3682
0.1953
0.3504
0.2399
0.3875
0.1501
0.3059
0.1338
0,2900
NENN
0.2776
0.4204
0,2068
0.3610
0.2777
0.4224
0.1658
0.3207
0.1361
0,2948
BERT
0.2582
0.4017
0.1863
0.3407
0.2540
0.4001
0.1608
0.3156
0.
0,3371
BERT+MaxSAGE
0.3028
0.4424
0.2128
0.3661
0.2859
0.4299
0.1687
0.3236
0.1828
0.3399
BERT AcIn
0.2705
0.4145
0.2024
0.3572
0.2527
0.4004
0,1572
0.3129
0.1849
0.3418
BERT+GIN
0,2790
0.4212
0,2040
0.3583
0.2613
0.4073
0.1611
0,3158
0.1858
0,3413
GraphFormers
0.2998
0.4393
0.2111
0.3642
0.2852
0.4294
0.1671
0.3220
0.1833
0.3405
BERT
0.2025
0.3552
0.1577
0.3163
081822
0.3361
0,1007
0,2603
0.1232
0.2845
BERT+NENN
0,3087
0.4470
0,2193
0,3719
0,2956
0.4382
0.1737
0,3280
0.1759
0,3341
Edgeformer-N
0.3206
0.4574
0.2320
0.3838
0.3106
0.4514
0.1849
0.3385
0.1944
0.3508
3399%
"2.3%
5.8%%
"3.2%
55.1%
33.0%
6.4%
3.2%
4.6%
"2.6%
A.5 NODE CLASSIFICATION
The 26 classes of Amazon-Apps nodes are: "Books & Comics", "Communication" "Cooking", "Education", " "Entertainment" "Finance", "Games", "Health & Fitness". "Kids", "Lifestyle", "Music". "Navigation". "News & Magazines", "Novelty", "Photography". "Podcasts". "Productivity": "Refer- ence", "Ringtones", "Shopping", "Social Networking", "Sports", "Themes", "Travel", "Utilities", and "Weather".
"Mussi"""""""""""".
15

---

Figure 3: Effect of the dimension of initial node embeddings. Published as a conference paper at ICLR 2023
The 2 classes of Moazzon-Movie nodes are: "Movies" and "TV"..
A.6 HYPER-PARAMETTE STUDY
0.23
0.30
0.15
0.31
0.225
0.375
0.145
0.305
0:175
0.335
0.22
0.37
0.14
0.3
NDCG
MRR
0.215
0.365
0.135
0.295
0.33
0.205
0.21
MRR
NDCG
0.355
0.36
0.125
0.13
NDCG
MRR
0.29
0.285
0.165
NDCG
0.2
0.35
0.12
0.32
16
32
[
(a) Amazon-Apaa
(b) Chodreeds-Children
(c) SteckOverllow
0.24
0.22
0.39
0.14
0.35
0.3
0:21
0.22
0.36
0.35
0.16
0.18
0.2
NDGG
0.35
0.37
011
0.13
0.12
0.29
0.28
ND03
0.17
0.19
0.38
0.36
0.2
MRR
0.34
0.33
0.31
0.92
0.14
0.31
0.1
0.26
0.15
[N,(00]
[M,(0)]
[M,(
(a) Amazon-App Apps
(b) Goodreads Chiilldee
(c) StackOverflow
Figure 4: Effect of the sampled neighbor size (i.e., (N))
Node Dimension. We conduct experiments on Amazon-Apps, Goodreads-Childmee and Stack- Overflow to understand the effect of the initial node embedding dimension in Eq. We test the performance of Edgeformer-N N on the link prediction task with the initial node embedding dimension varying in 4, 8, 16. 32, and 64. The results are shown in Figure where performance of Edgeformer-N generally increases as the initial node embedding dimension increases. This finding is straightforward since the more parameters an initial node embedding has before overfitting, the more information it can represent.
find that the
Sampled Neighbor Size. We further analyze the impact of sampled neighbor size for node rep- resentation learning Amazon-Apps. Goodreads- Chhildren, and StackOverflow, with fraction of edges randomly sampled for the center node. The result can be found in Figure We can find that the performance increases progressively as sampled neighbor size [Ne(v)] increases. It is intuitive since the more neighbors we have, the more information can contribute to center node learning. Meantime, the increase rate decreases (v)] increases linearly because the information between neighbors can have information overlap.
A.7 SELF-ATTENTION MAP STUDY
In order to study how the virtual node token will benifit the encoding of Edgeformer-EE we plot the self-attention probability map for a random sample in Figure We random pick up token from this sample and plot the self-ate attenttnnioo probability of how different tokens (x-axis), including virtual node tokens and the first twenty original text tokens, will contribute to the encoding of this random token in different layers (y-axis). From the figure, we can find that: In higher layers (e.g., Layers 10-11), the attention weights of virtual node tokens are significantly larger than those of original node tokens. Since virtual node token hidden states are Rxx2 and the original text token hidden states are of Aaxl (1 is text sequence length), the ratio of network tokens to text tokens is 2 in H (Eq where is the text sequence length. However, the self-attention mechanism can automatically learn to balance the two types of information by assigning higher weights to the corresponding virtual node tokens, so a larger number of tokens representing textual information will not cause network information to be overwhelmed.
A.8 REPRODUCIBILITY SETTINGS
For a fair comparison, the training objectives of Edgeformer-N and all the same. The hyper-parameter configuration for obtaining the results in Tables 1 and 2 can be found
baselines are
16

---

Published as a conference paper at ICLR 2023
Layer-wise Attention Map
1011
0.5
# Edgeformer Layer
-0,4
-0.3
1 2 3 4 5
0.2
-0.1
0
[node .]]
[token_]] []
[token_ 6]
[token_ .]]
[token 8]
[token 9]
[token_ 10]
[token 11]
[token_ 12]
[token_ 13]
[token_ moken_14, 14]
[token 15]
[token 16]
[token_]] 17]
[token_ 18]
[token 19]
[token_ 20]
[node
[token
[token
[token
[token
Tokens
Figure 5: Self-attention probability map of Edgeformer-E for random sample. The x-axis corre- sponds different key/value tokens and the y-axis corresponds different Edgeformer-E layers. In higher layers (e.g., layers 10-11), the attention weights of virtual node tokens are significantly larger than those of original node tokens. The ratio of network tokens text tokens is 2 / in H (Eq.4), where is the text sequence length. However, the self-attention mechanism can automatically learn to balance the two types of information by assigning higher weights to the corresponding virtual node tokens, so a larger number of tokens representing textual information will not cause network information be overwhelmed.
in Table 9 where "sampled neighbor size" stands for the number of neighbors sampled for each type the center node during node representation learning. This according to the average node degree of the corresponding node type. The edge classification and link prediction experiments are conducted on one NVIDIA V100 and one NVIDIA A6000, respectively.
determined
classifier. We employ the Adam optimizer (Kingmal Ba 2013) with the carly-stopping patience as 10 to train our classifier. The learning rate is set as 0.00T.
In Section 4.3
adopt logistic regression
A.9 LIMITATIONS
In this work, we mainly focus on modeling homogeneous textual-edge networks and solving funda- mental tasks in graph learning such node/edge classification and link prediction. Interesting future studies include designing models to characterize network heterogeneity and applying our proposed model real-world applications such recommendation.
17

---

Published as a conference paper at ICLR 2023
Table 9: Hyper-parameter configuration.
Parameter
Aaaazon-Move
Amazon-Apps Gooddreas--rrin
Goodreads
Coodrrads-Children Children
StackOverflow
optimizer
learning
le-5
weight decay
1e:3
Adam
le-8
early-stopping patience
training batch size testing batch size
100
node emhedding dim chunk k
12
max sequence length backbone PLM
64
BERT-buse--t unce-dd
sampled neighbor size
user:8 item:
user:3
reader:8
reader:6
expert:2
item:5
book:10
book:4
question:5
A.10 ETHICAL CONSIDERATIONS
While it has been demonstrated that PLMs powerful in language understanding (Devlin et al. 2019 Liu et al. 2019 Clark et al. 2020 there are studies pointing out their drawbacks such as containing social bias (Lang et al. 2021 and misinformation (Abid al. 202T). In our work,
focus on enriching PLMs' text encoding process with the associated network structure information, which could be way mitigate the bias and wipe out the contained misinformation.
18


Reviews:
[{"review_id": "zFJemHNDRS", "reviewer": "Reviewer_Sciz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors address the problem of graph representation learning where edges are associated with text. In doing so the authors propose to combine pretrained networks with GNN with local network aggregation to inject text representation of nodes to its neighbors. The authors conduct experiments on various datasets and tasks, along with visualization, to show the performance of the proposed method.\n", "summary_of_the_review": "\nSince the strengths are dominant, I would recommend 6: marginally above the acceptance threshold\n", "strength_and_weaknesses": "Overall this paper is clearly writing, well motivated. The method proposed is substantial and the results are convincing. \n\nThat said, there is still some issues that are better addressed:\n\n1. A lot of similar works have been researched on the graph with text attributes, like PLM-GNN and GraphFormers mentioned in the introduction. Although for them the text attributes are on the node, a straightforward adaptation of these methods to the current setting would be creating an extra node with text attribute for each edge. It's unclear how do they perform and compare with the proposed method\n\n2. In ablation study, are the differences statistically significant?\n\n3. in equation (4), since $z$s are of $R^d$ while $H$ is of $R^{d \\times n}$, are such concatenations \"balancing\"? (e.g. would $H$ dominates the final representation?)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ya8QmgQn8a9", "reviewer": "Reviewer_x3wZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose the method Edgeformers, which represents the text-attributed edges in a graph. \n\nThey utilize the pre-trained language model (BERT) and the concept of virtual node to represent each text in the edge with regards to its adjacent neighboring edges.\n\nSpecifically, in Edgeformer-E, they represent each edge attribute with its text and node embedding from their adjacent nodes using a transformer layer.\nIn Edgeformer-N, the node embeddings are represented as the aggregation of its adjacent edges’ embeddings from Edgeformer-E.\n\nIn experiments, they demonstrate the proposed method outperforms previous baselines in both edge classification and link prediction tasks.", "summary_of_the_review": "This paper has merit on its own in terms of how it affects learning edge representation on graphs with textual edge attributes.\n\nHowever, to fully demonstrate the validity of their method, more comparisons against recent edge-aware representation learning methods are required.\nAdditionally, more details on their approach in section 3 could make the writing clearer.\n\nAs a result, I give this paper a weak rejection.", "strength_and_weaknesses": "### Strengths\n\n- **Important problem and valid approach;** This paper addresses an important problem in edge representation learning where the text (document) is provided as an edge attribute. By reflecting graph structure into edge representations using pre-trained language models (PLM), the proposed method successfully handles the given problem.\n\n### Weaknesses\n\n- **Recent baselines are missing;** Relformer [1] is one of the advanced version of Graphformer, which is specialized for link prediction. Otherwise, EHGNN [2] has a similar approach which represents edges based on adjacent nodes and edges with dual hypergraph transformation. In my opinion, such powerful recent algorithms on edge representation learning should be compared as baselines for the proposed method.\n- **Hard to understand;** The suggested approach is challenging to comprehend. In order to comprehend the suggested method completely, I read the method section several times. Particularly, Figure 1 is hard to comprehend because there isn't a subscription for each component. Does the transformer layer of PLM have node information appended to it? If not, is Edgeformer another module that integrates the node information into the text representation after the PLM layers? It would be preferable, in my opinion, to incorporate some notations into Figure 1 or include additional, in-depth figures for each method.\n\n[1] Bi et al., Relational Graph Transformer for Knowledge Graph Representations, WWW 2022\n\n[2] Jo et al., Edge Representation Learning with Hypergraphs, NeurIPS 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0SRjAdwl0xM", "reviewer": "Reviewer_CnjZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "EdgeFormers is a new deep learning architecture to deal with networks with textual edges (e.g., email network or product reviews). Actually, it integrates two sequential modules: the first module (Edgeformer-E) aims at learning edge text-aware representations, the second module (Edgeformer-N) aims at learning node representation based on the edge representations of its neighbors. This work doesn't deal with attributed nodes, even though it should be easily adapted to do so. The main point is to go beyond considering the information associated to edges as a set of features, but treating it as a full text by the mean of Transformer-like mechanisms (multihead attention). EdgeFormers is trained on both edge classification (for Edgeformer-E module) and link prediction (for Edgeformer-N module). It's favorably compared to several recent architectures on 5 datasets.", "summary_of_the_review": "The proposed architecture is not fundamentally novel but it seems more fitted to deal with networks with textual edges. The model is quite well designed, even though some parts should be made clearer (relation between h_vi and z_vi, see above). EdgeFormer leads to slightly better results on 5 datasets, which finally explains my recommandation.", "strength_and_weaknesses": "Strength:\n\n- dealing with the textual information in a proper, modern way\n- experiments on various and quite big datasets with positive, consistant results\n\nWeaknesses:\n\n- The architecture is quite complex with multiple mechanisms that are combined and it's not always easy to figure out what's really going on in there. For instance, Edgeformer-E integrates node vectors noted z_vi (for vertices vi) in order to contextualized the edge vector h_eij. However, Edgeformer-N uses another node vector noted h_vi which is the aggregation of the surrounding h_eij. In this end, I wonder what's the relation between both types of vectors. I guess that z_vi is a *first version* that has been a priori initialized with another node embedding technique, and h_eij a *second (and better) version* that is maybe more contextualized.\n- Some recent works have pointed out the bias of GNNs that tend to \"oversmooth\" the information because of the propagation mechanism (see for instance the discussion here: https://towardsdatascience.com/over-smoothing-issue-in-graph-neural-network-bddc8fbc2472). It seems to me that EdgeFormer has the same king of limitation. If I'm correct, I think the authors should discuss this point and test on other datasets to clearly show when their model is applicable.\n- Even though the authors make some effort for reproducibility, I don't see any information related to the usual train/test procedure we expect in a machine learning paper.\n- It seems to me that the framework is transductive. What would be its ability for more inductive tasks? For instance, what about a new node (e.g., a new user) that posts a review on a product which is in the network? Is EdgeFormer able to deal with such a case?\n\nTypo:\n- I think \"z_i\" and \"z_j\" should be replaced by \"z_vi\" and \"z_vj\" in Eq.12 ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster