StarGraph: Knowledge Representation Learning based on Incomplete Two-hop Subgraph
Authors: Hongzhu Li
Xiangrui Gao
Linhui Feng
Yafeng Deng
Yuhui Yin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Knowledge Representation Learning
Knowledge Graph Embedding
Knowledge Graph
Self-Attention Network

Abstract: Conventional representation learning algorithms for knowledge graphs (KG) map each entity to a unique embedding vector, ignoring the rich information contained in the neighborhood. We propose a method named StarGraph, which gives a novel way to utilize the neighborhood information for large-scale knowledge graphs to obtain entity representations. An incomplete two-hop neighborhood subgraph for each target node is at first generated, then processed by a modified self-attention network to obtain the entity representation, which is used to replace the entity embedding in conventional methods. We achieved SOTA performance on ogbl-wikikg2 and got competitive results on fb15k-237. The experimental results proves that StarGraph is efficient in parameters, and the improvement made on ogbl-wikikg2 demonstrates its great effectiveness of representation learning on large-scale knowledge graphs.

Under review as a conference paper at ICLR 2023
STARGRAPH:
KNOWLEDGE REPRESENTATION LEARNING BASED ON INCOMPLETE TWO-HOP SUBGRAPH
Anonymous authors Paper under double-bling review
ABSTRACT
Conventional representation learning algorithms for knowledge graphs (KG) map each entity to unique embedding vector, ignoring the rich information con- tained in the neighborhood. We propose a method named StarGraph, which gives novel way utilize the neighborhood information for large-scale knowledge graphs to obtain entity representations. An incomplete two-hop neighborhood subgraph for each target node is at first generated, then processed by modified self- celf-attention network to obtain the entity representation, which is used to replace the entity embedding conventional methods. We achieved SOTA performance ogbl-wik wibkkk2 and got competitive results on fb15k-237. The experimental re- sults proves that StarGraph is efficient parameters, and the improvement made ogbl-wikikge demonstrates its great effectiveness of representation learning on large-scale knowledge graphs
INTRODUCTION
A Knowledge Graph (KG) is directed graph with real-world entities as nodes and relationships between entities as edges. In this graph, each directed edge together with its head and tail entities forms triple (head entity, relation, tail entity), indicating that the head and tail entities are connected by a relation.
Knowledge graph embedding (KGE), also known knowledge representation learning (KRL), aims to embed entities and relations into low-dimensional continuous vector spaces to characterize their latent semantic features. A scoring function is defined to measure the plausibility for triples in such spaces, then the embeddings of entities and relations are learned by maximizing the total plausibility of the observed triples. These learned embeddings can be used to implement various tasks such as knowledge graph completion (Bordes et al.. 22033] Wang et (014)), relationship extraction (Riedel al., 2013), entity classification (Nickel et al. 2001 etc. The plausibility of each triple is calculated on the embeddings of the entities and relations in it, and the embeddings are directly taken out from the embedding tables. Such shallow lookup decides that those models are inherently transductive. Moreover, the rich contextual information contained in the neighboring triples is not taken into account.
Compared with shallow embedding models, methods that are able to encode neighborhood informa- tion. usually perform much better across various KG datasets (Zhang & Chen 2018 Zhang al., Wang et al. 2019). Any generic graph neural networks could be employed as the encoder. However, there is a problem adopting these methods to large-scale knowledge graphs, for previous work (Nathani al. 2019; Wang al., 2020) takes the multi-hop subgraph of the node input. Due to the large number nodes and edges, multi-hop subgraphs in large-scale graphs can easily exceed the size limitation, and the subgraphs generation and network calculations can both be very time-consuming.
2021
The neighborhood surely contains information for the target node, therefore can be used for learning its representation. order to adopt neighborhood neural encoders in large-scale KG, an intuitive idea is to utilize partial neighborhood information instead of the complete multi-hop subgraph.

---

Under review as a conference paper at ICLR 2023
(a) complete 2-hop subgraph
(b) incomplete 2-hop subgraph
Figure 1: Illustration of subgraph generated by StarGraph. Dots and lines represent nodes and edges in the graph, respectively, with larger dots indicating anchors. The color red indicates the example target node and the sampled subgraph.
In this paper, we propose to learn the knowledge representation for each target node based its incomplete 2-hop neighborhood subgraph. Neighbors out of reach within two hops are not as closely related the target node according to us, complete 2-hop subgraph is given in Figure[[a), where we can see, even in such a small knowledge graph, the 2-hop subgraph comprises quite few nodes and edges and seems contain lot of redundant information. It is more efficient to construct a proper incomplete subgraph with a few nodes and edges.
taken into consideration. An example of
We got inspiration from the anchor-based strategy (Galkin et al. 2022). which selects a small frac- tion of all the graph nodes as anchors and learn embeddings only for anchors instead of all nodes. In our work, we sample anchors from the 2-hop neighborhood, along with the edges to reach each anchor, to construct the incomplete 2-hop subgraph, which is illustrated in Figure [b). The incom- plete subgraph is not restricted to contain only anchors, we can also sample some general nodes into the subgraph as supplementary information.
In order to reasonably model the subgraph structure and enable sufficient interaction of node em- beddings, we adopt a self-attention network extract the neighborhood information. Taking the characteristics of knowledge graphs into consideration, we modify the attention module to be more efficient and propose a novel way to embed the edges in a graph.
Comparing the subgraphs in Figure aa) and (b), the trimmed subgraph is much more efficient and is likely to more effective to describe the target node, especially for large-scale knowledge graphs. And this is demonstrated by our experimental results. The way node being represented by the incomplete subgraph is like how to locate star in the sky, which is to use a few bright stars (anchors) indicate the location. For the incomplete subgraph in the entire graph looks like constellation among the stars, we call the proposed method StarGraph.
RELATED WORK
Distance-based based models consist major branch of knowledge graph embedding methods. TransE (Bordes et al.] 2013) embeds relations and entities into the same vector space, and the relation embedding is interpreted as a translation from the embeddings of the head entity to the tail entity. RotatE (Sun et al. 2018) uses the rotations vectors to explain various relations, where the inverse relations can be modeled by complex embeddings. PairRE (Chao et al. 2021) propose to learn two embeddings for each relation, respectively used to map head and tail entity embeddings into the corresponding relation space. TripleRE (Yu al. 2021 learns another embedding for each relation on the basis of PairRE, and the extra relation embedding is used as a translation vector between the mapped entity embeddings of head and tail.
2

---

Under review as a conference paper at ICLR 2023
Contextual information has been widely used for knowledge graphs to enhance model perfor- mance, but in many different ways. CoKE (Wang et al. 2019) treats link prediction as a natural lan- guage processing (NLP) task, and trains the NLP model with long tltity-rllaioon-equeeee -equuuueeeessssssssss taken from the graph, linking entities originally distant from each other more closely. GC-OTE (Tagg et al.) 2020) directly integrates the graph context into the distance scoring function by considering tail/head entities with the same head/tail entity and relation all at once and minimizing the distance between their embeddings. NodePiece (Galkin et al. 2022) borrows the idea of subword-tokenizatie from NLP and proposes to use a small number of nodes as anchors, constructs vocabulary for each en- tity using surrounded anchors and relations, which implicitly includes contextual information, and calculates node representations from this.
Graph encoders can be conventional graph neural networks, e-g. GCN Kipp Welling 2017) and GraphSAGE (Hamilton et al., 2017), or the recently proposed Graph Transformer (Dwivedi Bresson 2021) and Graphormer (Ying et al. 202T), whose work are developed on the basis of Transformer (Vaswani et al. 2017). The basic unit in Transformer is the self-attention module, which provides a novel way to deal with the inputs. takes a sequence of feature vectors, (tokens as being called) inputs and updates each token with all others within the sequence, which is controlled with an attention matrix. It provides an effective way for information exchange and allows the tokens staying unordered, which seems suitable to process the structure of graphs.
The graph attention networks (Nathani et al. 2019) and graph attenuated attention networks (Wang et al. 2020 base their work on ConvVB (Dai Quoe Nguyen al., (018) and CapsE (Vu et al., 2019
which are proved invalid by Sun et al. (2020), thus will not be compared with in this paper.
METHODOLOGY
The StarGraph is a method to obtain entity representations based on incomplete neighborhood infor- mation, which includes two stages, generating and encoding subgraph for each node. The obtained entity representations, together with the relation embeddings, are used to compute the scores of the triples by distance-based score function, and optimized with the self-adverse negative sampling loss (Sun et al. 2018). We will describe each part in detail below.
3.1 SUBGRAPH GENERATION
To generate the proper incomplete subgraphs, we should start with select part of the nodes to form the anchor set. For ease of description. we denote the anchor set as A, all nodes of the total graph as N. There are A E N, and |A| S |N]. Then for each target node, we pick part of the anchors and nodes within its 2-hop neighborhood to construct the subgraph.
3.1.1 ANCHOR SET
At first, a minority of nodes are selected as anchors. When the number of anchors is fixed and much less than the number of graph nodes, the most intuitive idea is that the anchors should be selected according to the degree of centrality in order to provide common information to as many nodes as possible. Following this idea. Galkin et al. (2022) employs the deterministic anchor selection strategy where 40% of the total number of anchors are nodes with top Personalized Page Rank (Page et al. 1998) scores, 40% are top degree nodes, but remaining 20% are selected randomly for they found random anchor selection to be as effective as centrality-based strategies. We argue that the random part introduce the uncertainty, and anchor sets generated by the same strategy may have different performances in experiments. Besides, there is no need to use two centrality measures, and we only use the degree of nodes as the measure.
If the selected anchors are too concentrated, it may result in that some nodes are surrounded by too many anchors while some other nodes have no anchors around them. To alleviate this problem, we introduce a hlpper-parameter skip-threshold. When the percentage of anchors in the neighboring nodes of a node exceeds the threshold, that node will not be selected as an anchor. We also need to specify the anchor-set-size i.e., how many nodes are selected anchors. The process of generating the anchor set can be seen in Algorithm [I] Detailed discussion on the effect of skip- tiip-threshold can be found in Appendix A but in the main text the value is fixed at 0.5.
3

---

Under review as a conference paper at ICLR 2023
Algorithm/ 1: Generate the anchor set for the graph
Input: graph nodes N, skip threshold, anchorset size
Output: anchor set A
+- sort or Secereeeeeecciiiggneeoeooeeegror
anchor_set +-
for node & sorted_nodes do
'bbors + wigh./oorpeegbbbrrreboooooo anchor_nbor t nbors n anchor _sct
if size(anchor sooncoornnnborsssssschbooss skip_three thresh then
continue end
// many anchors around, skip this node
anchor + anchor U {node} sizeeanchor__set 2 anchor setiooreessseee then
10
11
break
// got enough anchors, end the loop
12
end
13 end
(a) 2-hop subgraph
(b) BFS
(c) balanced
Figure 2: Six anchors sampled from the 2-hop subgraph of the target node @ adopting different strategies. Dots and lines represent nodes and edges, where colored dots anchors and solid lines indicate how we reach the anchors. The numbers on dots indicate the sampling order of anchors.
3.1.2 ANCHIORS SAMPLING
For each node, fixed number of anchors sampled from its 2-hop neighbors, together with the paths to reach the anchors, to construct the incomplete subgraph. When the anchors within two hops more than needed, for example, sampling six anchors for the target node from its 2-hop subgraph given in Figure only the first part of the anchors sampled will be retained. The easiest sampling order to think is treadth-first-seacch (BFS), but it might induce severe imbalance in anchors distribution, as shown in Figure ([.... The sampled anchors are concentrated around node#2, and this sampling strategy fails to associate the target node with some non-anchor neighbors.
We propose a more balanced sampling strategy that aims to share more anchors with the non-an won-anchor neighbors, by sampling one-hop anchors of the target node and its every non-ane con-anchor neighbor turn. The detailed steps are described in Algorithm 2 and the sampling results of this strategy are shown in Figure Cc. Note that anchors with smaller degrees will be selected preference according to line#7 and #13 in Algorithm 2 It is based on the considering that 1) less-popular anchors make each target node more recognizable from others; 2) less-popular anchors can be visited more often to alleviate the long-tail distribution problem in the usage of anchors.
3.1.3 NEIGHBORS & CENTER
Neighbors refer to fixed number of nodes sampled from one-hop neighborhood of the target node, and center refers to the target node itself. We use these nodes as additional information to further enrich the description of the target node. Different from anchors, that picked from the predefined anchor set, neighbors and centers could be any node in the graph. We sample the neighbors in degree-decreasing order, trying them make common description for each target node.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 2: Sample anchors for subgraph of each node
Input: anchor set A, node, sample size
Output: subgraph anchors
Def one Anphossssne as one onneHopNeighborsswodee n anchor_sed
non_anc_nbors (oee:oopNeiillboosssneeee) onc/Hoppncchors(nodd)
subgraph.anchin t- 0
while size( wwwbrraphhncchoooe
< sample_size do
remaining Lancs t- one one wow.Hooonnocossoooolo - subgraph. subgraph.anchop anchors if remaining ,ncs + 9 then
sariiinnnnnnssgeeeenneeiee nexl anc +- alddeWihhMiiinnnii Deeeareeeeeeeeeeereieiiiii ancs) subgraph.ancherr end 10
subgrapp..nchhhne aanchors U (nexlanc))
for nbor E non_anc..bort do remaining. _ancsnggnnssssce +- oncHHopAnchhrs((nbe)) if remaining.ovv anns & @ then
catigrrappannchrrr
12
nexl_anc MendeW/WWWiinnnnneerrrreereerr
subgraph.anchors + subgraph.anchor U [next_onc) end 15
end 16
17 end
3.2 SUBGRAPH ENCODING
self-attention network (Vaswani al., 2017), whose input is considered sequence though, its operation on each element within the sequence is naturally global and undifferentiated. Therefore, we consider perfect choice encode graphs. The network takes the embeddings of nodes within generated subgraph as inputs, each node treated as token. Since a self-attention layer has global receptive field, one layer can adequately mix the information of nodes. We use single transformer block, i.e. one attention layer followed by two linear layers, to construct the network. After this, the output node features are aggregated by mean-poot mean-pooling across the subgraph to get the representation of target node. More discussion on the choice of encoders can be found in Appendix B
3.2.1 REDUCED ATTENTION MODULE
The attention module consumes a lot of computational resources, ficient way to implement the attention mechanism. For a standard k-heads attention layer, it takes X E RixDundee
try to explore more ef-
input, sets the dimension of each head as Dhead = Dhidden k,, and gets the output Y 1,,,,,,, Y2..... Ya] E R Dieaden by concatenating the outputs of k heads together. The output of the i-th head Y; RLxDDeedd is calculated by
XWQ. X
Y; softmax
XWV.
DDhead
where Wo., K. Wv. E RDisadnn Dread are model parameters to be learned.
The role the attention module to integrate information from all nodes the subgraph. Ensuring this, make two modifications to attention head to reduce the amount of computation. Firstly, to get a usable attention matrix, the second dimension of WQ, and W,, is not forced to equals Dread, setting smaller dimension Dattn < Dhead will reduce the computational complexity. Secondly, with an MLP afterward, there is no need to transform the features in the information mixing step. directly slice X into k parts with the same shape of XW v. to
Since Dread Dhidden/ k, omit the multiplication operation of X and W Wv... The output of the i-th head Y; RL. Dreed of reduced attention layer is calculated by
XW W.a. WW, XT
Y, = softmax(
Xi,
(2)
Dattn
5

---

Under review as a conference paper at ICLR 2023
where X; E RhexD are directly taken out from X DDiideen Daten are model parameters to be learned.
[2, .x]l, and WQ.., WK,
Beside the attention layer, the computational complexity of MLP can be reduced by decreasing the intermediate dimension Dridden * mlp_ratio. The default value of hyper-parameter mlp-ratio is 4, and a smaller value can be considered.
3.2.2 PATH INFORMATION FUSION
Treating each anchor token, the anchor path, i.e., edges the anchor needs pass reach the target node, should be considered as the position information. Instead of following the practice of previous work, where the position embeddings are directly added onto the token embeddings, we propose to take the characteristies of KGE into consideration. Insprired by the score function of TripleRE (Yu et al. 2021).
f+( = h (r + 1) to (r' + +1) +r|l
(3)
based the assumption that
o (r' + 1) (r" + +) +r,
(4)
simplify the conditions and assume that
t (r" +r.
(5)
Substituting "head, relation, tail" with "anchor, path, token", respectively, we can get token embed- ding by adding path information to 1-hop or 2-hop anchor as
anchor o (Pf + 1) + p
when path = {pi),
token = Janchor (pf + 1) +pi (pC + +
(6)
when path = {P1,P2).
Note that after incorporating the path information, the token embedding is more like the represen- tation of the target node rather than the original anchor, therefore mean rean-pooling is proper way to aggregate information from the tokens.
3.2.3 SUPPLEMENTARY NODES
As described in section 3.1.3 a subgraph may contain nodes sampled from N, providing additional information for the target node. As with the entire graph, each node keeps some unique information, while only a small group of nodes are anchors, carrying much general information. Denoting the embedding sizes of A and N as D and Dv. set a large Da improve the information capacity, and a small D to reduce the probability of overfitting. To get the tokens of the same embedding size, D is used Diidden the hidden size for attention module, and the node embeddings mapped to dimension DA by linear layer ahead.
Type embedding is added the token embedding before the attention module, indicating each node within the subgraph to be the type of anchor, neighbor or center.
3.3 SCORE FUNCTION
We adopt a score function that modified upon TripleRE (Yu et al. 2021). recently proposed distance-based method. The formula of TripleREv2 is
th, -| (r" a-ulh to (r' + u 1) +r|l
where o denotes the element-wise product, and t correspond to the head and tail entity embeddings respectively, Ir', r', .r] comprise the relation embedding, is a constant specified by users, set in the original work. We modify equation (7) by switching 1 with r" r"/r', so that the coefficient acts on the relation embedding. Now the score of triplets (h, r) is calculated as
(h, t) = - l (u 1 + 1) to (u + 1) r|| = h t t u (hor" - to tor')
(8)
which can interpreted as a boosting method based on TransE and PairRE. The new form referred to as TripleRE' in the rest of this paper. Although TripleRE' is exactly the same TrvpleREv2 when u equals 1, according to our experimental results, TripleRE' usually performs better than TripleREv2 during tuning the hyper-parametes u.

---

Under review as a conference paper at ICLR 2023
3.4 OPTIMIZATION
We utilize the self-adversarial negative sampling loss (Sun et al. 2018) as objective for training,
explaft(hs;
L - log lo - fr( wi log (fx((,, 7),
wi
f=1 exp(af
where y is fixed margin and is the sigmoid function. We randomly sample negative triples for each positive one, and (h, tg) is the i-th negative triple.
4 EXPERIMENTS
4.1 DATASETS AND METRIC
ogbl-wikikge (Hu et al. 2020) is a large-scale knowledge graph extracted from the Wikidata knowl-
edge base (Vrandečié & Krötzsch| 2014), containing 2,500,604 entities, 535 relation types and 17.13..111 triplet edges (head, relation, tail). fb15k-237 (Toutanova & Chen 2015) is subset of fbI5k (Bordes et al. 2013), that consists triples from Freebase, with inverse relations removed. It contains 14,505 entities, 237 relation types and 310,079 triplet edges.
Task and Metric The task is to predict new triplet edges given the training edges. The evaluation metric follows the standard filtered metric widely used in KG. Specifically, each test triplet edges are corrupted by replacing its head or tail with negative entities. The goal is to rank the true head (or tail) entities higher than the negative entities, which is measured by Mean Reciprocal Rank (MRR).
4.2 DETAILED SETTINGS
We set batch-size to 512, negative-samplie sannive-sammlinn-sizee to 64, and train the model for 500,000 steps on oibi-2i and 100,000 steps fb15k-237. The learning rate is initialized to 0.0001 unless otherwise stated, and decreased by 0.1 at half the maximum training step. Dropout adopted after each linear layer, and the drop-ratio is 0.05. AdamW is used as the optimizer in experiments.
For the generated incomplete subgraphs, anchors are always required. The paths and the supplemen- tary nodes are optional, which not included in subgraphs unless being specified. The number of anchors and neighbors (if used) sampled for each subgraph is fixed to 20 and 5 respectively, and any shortfall is filled with the [PAD] token. The default values for the embedding dimension DA 256, DN = 32. apper-parameter in equation (8) is tuned and set to 0.1 on ogbi-wikikg2 and 1.0 on fb15k-237, and y in equation (9) to 6.0 following PairRE (Chao al. 2021
4.3 ABLATION STUDY
The strategies for anchor set generation and anchors sampling are proposed in sec- tion 3.1.T and 3.1.2 respectively, and display the effects of them in Table [ Comparison be- tween the first two rows validates the effectiveness of the proposed sampling strategy by 1.5% gap. The results of second and third rows are close to each other, despite the uncertainty in degree+PPR+ raR+rand does not exist in generation strategy. Comparing the last two rows, the performance gain shows that a larger anchor set performs better, at least under the condition that [A] < NN]. For the following experiments, the marked in gray is used the default setting with anchors unless otherwise specified.
Anchors
Network The first two rows in Table show that, the larger dimension of features (also known as DA and hidden-dim in this paper) the better result, meantime the higher computational complexity. Comparing the overall performance of network 256-32-4 with its reduced counterpart 256-R32-4, we can see that the reduced attention module proposed in section 3.2.1 saves a little training time without degrading the results. With further adaptation of attn-dim and mip-ratio, network 512-R8-2 achieves a better trade-off between the results and the consumption training time. The row marked in gray is used as the default network for the following experiments, unless otherwise specified.

---

Under review as a conference paper at ICLR 2023
Table 1: The results on ogbl-wikikg2 with different anchor sets and anchor-sampling strategies.
Anchor Set
|A]
Sampling
Test MRR
Valid MRR
Pegree+PPR+randd Galkin al.
2022
20,000
BFS
0.6899
0.6953
degree+PPR+rand Gaakkn et al.
2022
20,000
balanced
0.7040
0.7104
degree-skip0.5 degree-skip0.5
20,000
balanced
0.7042
0.7093
80,000
balanced
0.7084
0.7164
Table 2: The results on ogbl-wikikg2 with different networks. The 'R' in Network ID indicates the
reduced' attention module, operating like equation (2). Train Time refers to the practical time spent for whole training process, and every experiment performed on single A A100 GPU.
Network ID
hidden-dim
attn-dim
mlp-ratio
Train Time
Test MRR
Valid MRR
256-32-4
256
32
4
28.7h
0.7042
0.7093
512-64-4
512
64
48.7h
0.7141
0.7189
256-R32-4
256
32
27.4h
0.7067
0.7085
512-R8-2
512
8
2
33.3h
0.7129
0.7146
Subgraph Components On the basis of anchors, other types of information are added to sub- graphs in turn. The results in Thble that the addition of both path and supplementary nodes information can make obvious improvement. The subgraphs containing all kinds of information have the best results overall.
4.4 RESULTS ON OGBL-WIKIKE & 1515K-237
conduct experiments, and present some of our best results Table and 5 with the corresponding experiment settings given in Appendix C showing the trade-off between the number of parameters and the results.
We combine the advantaged strategies and modifications in section 4.3
Our method achieves state-of-the-une results on oibl-wikikg2 with a significant 3%-ish improve- ment. We presume that the construction of incomplete multi-hop subgraphs is summarization and extraction of the basic concepts in a knowledge graph, and much information already stored in subgraph structures, making the learning task easier and thus leading better results, especially for large-scale knowledge graphs. On fb15k-237, our results are constrained by the distance-based model, for the parameters in StarGraph are optimized through the score function of TripleRE' So we directly train the embeddings of entities and relations with TripleRE' as the conduct group. StarGraph does not achieve as significant improvement on fb15k-237 as on ogbl-wikikg2 though, the results are still comparable with its conventional KGE counterpart with less parameters, which validates that StarGraph dose work on smaller KGs as well.
As with the number of parameters, the efficiency of StarGraph is more evident on large-scale KGs, as the network itself takes up a certain amount of parameters. Comparing the #Params marked in gray, the number of embedding parameters of StarGraph on fb15kk237 is only about 30% (2304 x 512)/(14505xx 256) of the control item, but the total number of parameters is about 80%; the number
Table The results on ogbi-wikikg2 with subgraphs containing different types of information. The i'ndicctes the corresponding type of information to be contained.
anchors
path center
neighbors
Test MRR
Valid MRR
0.7067
0.7085
0.7222
0.7373
0.7197
0.7235
0,7248
0.7307
0.7248
0.7319
0.7239
0.7403
8

---

Under review as a conference paper at ICLR 2023
Table 4:
The results oghl-wikikg2. Results of related work are taken from the OGB Leaderboard.
Model
#Dims
#Params
Test MRR
Valid MRR
TransE
500
1,250.569.500
0.4256+0.0033
0.4272+0.0030
RotatE
250
1,250,435,750
0,433240.0002
0.4353#0.0022
PairRE
200
500,334,800
0.5208+0.0027
0.5423=0.0022
AutoSF
500,227,800
0.5458+0.0000
0.5510=0.000
ComplEx
250
1,250,569,50
0.5027=0.00%
0.3759#0.00% 6,0016
TripleRE
200
500,763,337
0.5794+0.0000
0.6045#0.00% 00004
TripleREv2
200
500,763,337
0.6045+0.0017
0.6117#0.0008
ComplEx-RP
50
250,167,400
0.6392+0.0045
0.6561#0.0070
+ + NodePice
6,860,602
0.5703+0.0035
0.5806#0.004
TripleREv2 + NodePiece
200
7,289.002
0.6582+0.0020
0.6616#0.001
TripleREv3 + NodePiece
200
36,421,802
0.686640.0014
0.6955+0.0008
InterHT NodePiece
200
19,215,402
0.6779+0.001s
0.6893#0.0015
TranS + NodePiece
200
19,215,402
0.6882+0.0011
0.6988#0.0006
Tlange) NodePiece
38,430,804
0.6939#0.0011
0.7058#0.0018
256
7,148,802
0.7222
0.7373
StarGraph + TripleRE'
512
44,819,970
0.7263
0.7407
512/32
93,039,522
0.7290
0.7327
Table The results on fb15k-237. Results of related work are taken from the corresponding papers, despite TransE being provided by Dai Quoe Nguyen et al. (2018).
Model
#Dims
#Params
Test Hit 10
Test MRR
TransE
100
1,474,200
0.465
0.294
RotatE
1000
29,484,000
0.533
0.338
PairRE
1500
22,468,500
0.544
0.351
GC-OTE
400
0.550
0.361
CoKE
256
2 10,190,000
0.549
0.364
NodePiece + RotatE
200
x 3,200,000
0.420
0.256
256
3,895,296
0,5429
0.3455
TripleRE'
1000
15,216,000
0.5520
0.3514
512
3,148,299
0.5454
0.3426
StarGraph TripleRE'
512
4,275,714
0.5475
0.3459
of of embedding parameters of StarGraph ogbl-wikikg21 is 1% (20000 205)//2200000 x 200) of the control item and the total number of parameters is only 1.4%.
5 CONCLUSION
In this paper, we propose StarGraph, novel method to learn knowledge representations by generat- ing and encoding the incomplete 2-hop subgraph for the node. The experimental results verify that our method is parameters-efficient and can obtain better or comparative results with fewer parame- ters on different datasets. More importantly, StarGraph achieves significant improvement of results on large-scale knowledge graphs.
Based upon the core idea of incomplete subgraph, we have proposed several strategies and modi- fications for the implementation. Though each strategy and modification is proved to be effective alone, it is worth further exploration on how to make them work collaboratively to the best effect and whether there also able to perform other related tasks, such as inductive link prediction, node classification. etc. All these treated future work.
better alternatives. Additionally, beside the transductive link prediction, StarGraph

---

Under review as a conference paper at ICLR 2023
REFERENCES
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data, Advances neural information pro- cessing systems, 26, 2013.
Linlin Chao, Jianshan He, Taifeng Wang. and Wei Chu. Pairre: Knowledge graph embeddings via paired relation vectors. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4360 4369,,36,4000 2021.
Tu Dinh Nguyen Dai Quoe Nguyen, Dat Quoc Nguyen, and Dinh Phung. A novel embedding model for knowledge base completion based convolutional neural network. In Proceedings of NAACL-HLLT, pp. 327-333, 2018.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks graphs.
AAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2021.
Mikhail Galkin, Etienne Denis, Jiapeng Wu, and William L. Hamilton. Nodepiece: Compositional and purameter-efficiien representations of large knowledge graphs In International Conference on Learning Representations, 2022.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs Advances in neural information processing systems, 30, 2017.
Weihua Hu, Matthias Fey, Marinka Zitmik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 18 221333,, 2020.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR). 2017.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. Learning attention- based em- beddings for relation prediction in knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 47 4710-4723, 2019.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In leml, 2011.
I. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Stanford Digital Libraries Working Paper, 1998.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 74 84, 2013.
Zhiqing Sun, Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representationss 2018.
Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha Talukdar, and Yiming Yang. A re- evaluation of knowledge graph completion methods. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5516-5522, July 2020.
Yun Tang, Jing Huang, Guangtao Wang, Xiaodong He, and Bowen Zhou. Orthogonal relation transforms with graph context modeling for knowledge graph embedding. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 2713-2722, 2020.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality pp. 57-66, 2015.
10

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communi- cations of the ACM, 57(10);78-85, 2014.
Thanh Vu, Tu Dinh Nguyen, Dat Quoe Nguyen, Dinh Phung, et al. A capsule network-based embedding model for knowledge graph completion and search personalization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2180-2189, 2019.
Quan Wang, Pingping Huang, Haifeng Wang, Songtai Dai, Wenbin Jiang, Jing Liu, Yajuan Lyu, and Hua Wu. Coke: Contextualized knowledge graph embedding. arXiv:1911.02160 2019.
Rui Wang, Bicheng Li, Shengwei Hu, Wen Wen Du, and Min Zhang. Knowledge graph embedding via graph attenuated attention networks. IEEE Access, 8:5212-5224, 2020.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by trans- lating hyperplanes. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 28, 2014.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Yan Liu. Do transformers really perform badly for graph representation? Advances Neural Information Processing Systems, 34:28877-28888, 2021.
Long Yu, ZhiCong Luo. Deng Lin, HongZhu Li, Huan Yong Liu, and YaFeng Deng. Triplere: Knowledge graph embeddings via triple relation vectors. viXra preprint iXra:2112.0095 2021.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang. and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems, 34:9061-9075 2021.
11

---

Under review as a conference paper at ICLR 2023
Table 6: The results ogbl-wikikg2 with different anchor sets.
thi-tthreshold
anchor-set-size
anchor-ratio
[A]
Test MRR
Valid MRR
1.0
20,000
0.008
20,000
70.88
71.09
0.5
20,000
0.008
20,000
70.67
70.85
0.2
20,000
0.008
20,000
70.65
70.85
1.0
80,000
0.032
80,000
71.01
71.30
0.5
80,000
0.032
80,000
71.12
71.53
0.2
80,000
0.032
79,157
71.03
71.66
Table 7: The results on fb15k-237 with different anchor sets.
threshhreehold
anchor-set-size
anchor-ratio
[A]
Test @it@ 10
Valid Hit@ 10
1.0
1,450
0.1
1,450
.53.55
54.15
0.5
1,450
0.1
1,450
53,97
54.58
0.2
1,450
0.1
1,450
53.80
54.43
1.0
14,505
1.0
14,505
54.33
54.93
0.5
14,505
1.0
4,503
54.75
55.06
0.2
14,505
2,304
54.54
55.12
A EXPERIENCE ON ANCHOR SET GENERATION
During the generation of anchor set, the hyper-paramete skip-threshold is used to control the spar- sity of anchors, the smaller value the sparser anchors; anchor-set-size decides the maximum number of anchors, which also be with anchor-ratis the proportion of anchors occupying in the graph. Note that when skip-threshold is small, skipping too many nodes may cause the actual size of the anchor set |A| to smaller than the specified anchor-set-size
The experimental results with different values of skip-threshold and anchor-set-size are displayed Table [ 7 where we can summarise some experience as follows. When the skip skip-threshold is fixed, the larger |A], the better results. But when the skip-threshold varies, the situation becomes little tricky. Comparing the results on egbl-wikikg" when skip-threshold=200000 we can draw the conclusion that when the anchor ratio is extremely small, high skip-threshold to keep the best few known nodes as anchors should bring better results. But when the anchor set exceeds certain size, such 80,000 ogbl-wikikg2 and 2,000+ fb15k-237, lower skip-threshold leads to better results with an equal or even smaller [A]. Anyway, a proper skip-threshold helps to make a better trade-off between the number of parameters and the results.
B EXPERIMENTS WITH DIFFERENT ENCODERS
To explore the effectiveness of self-attention network to encode graphs, we conduct the control experiment of different encoders and present the results in Table MLP consists of two linear layers and non- non-linear activation between them, Attention+Mean is the encoder adopted in StarGraph, and Mean omitting the attention module just mean-pooling operation performed over all tokens. Note that the case marked in gray is actually the same method as NodePiece, in spite of the mlp-rutio here instead of 2 in their experiments.
Table The Test/Valid MRR on ogbl-wikikg2 with different encoders different situations,
Sampling
path
MLP
Mean
Attention+Mean
BFS
0.6886/0.6924
0.6749/0.6733
0.6899/0.6953
balanced
0.7079/0.7152
0.6596/0.6587
0.7067/0.7085
balanced
0.7094/0.7247
0.7101/0.7250
0.7222/0.7373
12

---

Under review as a conference paper at ICLR 2023
Table 9: Experiment settings for StarGraph ogbl-wikikg2.
|A|
Sampling
Network ID
path nbors&center learning rate
Test MRR
Valid MRR
20,000
balanced
256-R32-4
le-4
0.7222
0.7373
80,000
balanced
512-R8-2
5e-5
0.7263
0.7407
20,000
balanced 512-R8-2
2e-4
0.7290
0.7327
Table 10: Experiment settings for StarGraph on fb15k-237.
|A]
Sampling
Network ID
path nbors&center
learning
Test Hit@ 10
Test MRR
2,304
balanced
512-R8-2
5e-4
0.5454 0.5475
0.3426
4,503
balanced 512-R8-2
5e-4
0.3459
Compare results with different unchors-sampling strategies, the balanced sampling proposed by us achieves 1%+ improvement for both MLP and StarGraph. But when it comes the introduction of path information, MLP does not process the subgraphs as effectively StarGraph. By the way, it is interesting to compare the results of Mean with or without the path information. The proposed infu- sion way of path information, which is developed form the KG distance-based model mentioned section 3.2.2. are proved to effective for the graph structure by the 5%+ improvement.
Overall, the experimental results confirm that the self-attention network cooperates the best with our subgraph-based representation learning.
C IMPLEMENTATION AND HYPER-PARANEN
The experiment settings for StarGraph Table 4 is given in Table 9 For each setting, the best learning rate is chosen from (5e-5, le-4, 2c-4) according the experimental results.
For the results Table 5 model TripleRE' corresponds to the conventional KGE method, the mension of embeddings is provided #Dim, and the best learning rate is 2e-4 in {1e-4, 2e-4, 5e-4, le-3}. The experiment settings for StarGraph is given in Table and the learning rate chosen from (le-4, 2e-4, 5e-4, le-3).
The selection the feature dimension and the combination of different modifications are in fact restricted, mainly due the video memory size. Note that adopting multiple advantaged strategies and modifications does not always result the sum improvements, and the best learning rates for different networks different sizes of anchor are usually different. It will take lot of work in the future to explore more of the potential for StarGraph.
13


Reviews:
[{"review_id": "sKNfZ9dD7O", "reviewer": "Reviewer_AAWP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new knowledge representation method named StarGraph, which serves as an encoder and aims to generate high-quality entity embeddings for downstream score functions. The proposed StarGraph mainly consists of two stages: subgraph sampling and encoding, and the authors manage to optimize the efficiency in both stages. Experiments demonstrate that StarGraph achieves a new SOTA result on a large-scale knowledge graph benchmark (wik90m-v2), while being more parameter efficient than other baseline models.", "summary_of_the_review": "This paper proposes an effective method to achieve new SOTA results on large-scale knowledge graph completion. Although the performance is strong, some concerns about motivation and ablations should be carefully addressed before this submission can be published. \n\n===============Update After Rebuttal===============\n\nI have read the author's response (the authors did not upload a revised submission). However, my concerns about the motivation and methodology are not addressed. The author fails to provide sufficient evidence to explain why we must design the model as they do now. In my view, this submission looks more like a technical report than an academic paper. Thus, I am leaning to reject it.", "strength_and_weaknesses": "## Strength\n1. The idea of using subgraphs around entities to generate expressive entity embeddings is reasonable.\n2. The authors propose a bag of tricks to reduce the computational overload for different steps of knowledge representation learning.\n3. The proposed StarGraph significantly outperforms previous SOTA models on the wiki90m-v2 dataset.\n4. Detailed experimental settings and codes are provided, so the reproducibility looks nice.\n\n\n## Weaknesses\n1. **Motivation**  \n    1.1 Why we can use incomplete subgraphs is not well motivated. In the introduction, the authors use Figure 1(a) to demonstrate that complete subgraphs contain a lot of redundant information. However, is there any evidence that this information is redundant? Obviously, the completed subgraph in Figure 1(a) contains more topology information than the incompleted subgraph in Figure 1(b). Although using incomplete subgraphs may lead to competitive results, we can not tell which one is more suitable for knowledge representation learning just through a toy figure illustration.   \n\n    1.2 Why we should use an anchor-based strategy is unclear. Even if using incomplete subgraphs is necessary for large-scale knowledge graphs, we can use a sample-based method (e.g., GraphSAGE) without anchors.\n2. **Related Work**  \n    The authors did a thorough review of related work. However, the differences between StarGraph and some existing works (e.g., NodePiece and GraphSAGE) are not fully discussed.\n3. **Methodology**  \n    3.1 The overall pipeline is unclear. I strongly suggest the authors provide a figure illustration for the overall architecture of StarGraph.\n\n    3.2 In Section 3.2, the authors claim that they use a single transformer block. I am wondering if it is a standard transformer block, i.e., is there a residual connection and layer norm in the used module?\n\n    3.3 The authors highlight two-hop subgraphs in the title. I am not sure whether two-hop is necessary. Can we use one-hop or three-hop subgraphs?\n\n    3.4 The path information is captured by TripleRE in Section 3.2.2. However, it seems that the proposed framework is irrelevant to the path encoders. Therefore, I suggest the authors try more KGE models when encoding paths.\n\n    3.5 I am curious about how to induce Equation (8) from (7). It would be better to provide a detailed derivation.\n4. **Experiments**   \n    4.1 It'd be better to put the main results before ablation studies.\n\n    4.2 The Network ID in Table 2 is confusing.\n\n    4.3 According to Table 3, the combination of anchors and neighbors already leads to the best test MRR, which demonstrates that path and center are unnecessary.\n\n    4.4 As the authors claimed, the main contribution of this paper is a new graph encoder. Therefore, the graph encoder should be able to improve the performance of different decoders not only the proposed TripleRE', e.g., TransE, RotatE, AutoSF.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lV7A3fGyNrQ", "reviewer": "Reviewer_keWz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Considering the current KG embedding methods of conventional shallow embedding models, which ignore the contextual information,   as well as the generic graph neural networks, which are difficult to expand to a KG with a large number of nodes and are very time-consuming, the authors propose a new knowledge representation learning method based on incomplete two-hop subgraph to leverage the neighborhood information with little computational complexity. ", "summary_of_the_review": "\nThe current version is not suitable for ICLR.", "strength_and_weaknesses": "The strength of the paper:\n\nIn order to save time as well as keep much information from the neighbor nodes,  the authors propose a new method based on an incomplete 2-hop subgraph for knowledge representation learning.  \n\nThe weakness of the paper:\n1. There is an error in the last two rows of the Table. 5 obviously.  \n2. The proposed model is not evaluated enough on different datasets and different tasks. In the paper, the authors only evaluate the proposed method on two datasets on one task.\n3. In fact, the proposed model is a variant of NodePiece and is not innovative enough.\n4. In my opinion,  different types of anchors sampling methods are needed to be investigated in the ablation study.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c2WE8UeEKDt", "reviewer": "Reviewer_ZZ4V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argue that Conventional method can not utilize  neighborhood representation effectively for large-scale knowledge graph. The paper propose STARGRAPH, a node selection strategy, to find the superior node in two-hop subgraph. The results show that the proposed method get SOTA results in large-scale knowledge graph link prediction.", "summary_of_the_review": "The proposed method is effective, but further reflection on KG is missing.", "strength_and_weaknesses": "\nStrength:\n\nExperiment on the ogbl-wikikg2 link prediction (Table 4) get SOTA results.\n\nWeakness:\n\nThe proposed method in the paper has limitations. The proposed method is to select better nodes in graph, which is a widely used idea [1][2][3]. Such an approach used on the knowledge graph lacks background and motivation. Also, as a KG-independent method, experiments on big graph are missing.\n\nTreating the knowledge graph as a simple directed graph ignores the semantic properties of the knowledge graph, and such work is not conducive to the development of knowledge graph research. For example, whether the selected nodes make important at the knowledge level, the experiment lacks a case study to demonstrate this.\n\nCompared to the previous approach [4], the proposed method experiments on knowledge graph related tasks and it is difficult to show that it is generalizable. \n\n[1]. Alsentzer E, Finlayson S, Li M, et al. Subgraph neural networks[J]. Advances in Neural Information Processing Systems, 2020, 33: 8017-8029.\n\n[2]. Wang X, Ji H, Shi C, et al. Heterogeneous graph attention network[C]//The world wide web conference. 2019: 2022-2032.\n\n[3]. Veličković P, Cucurull G, Casanova A, et al. Graph attention networks[J]. arXiv preprint arXiv:1710.10903, 2017.\n\n[4]. Galkin M, Wu J, Denis E, et al. Nodepiece: Compositional and parameter-efficient representations of large knowledge graphs[J]. arXiv preprint arXiv:2106.12144, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I7hZdcj50rT", "reviewer": "Reviewer_1SnH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a StarGraph method to learn entity representations for large-scale knowledge graphs. The StarGraph first generates an incomplete two-hop neighborhood subgraph for each target node by picking part of the anchors and nodes within the 2-hop neighborhood. Then a transformer network is applied to obtain the entity representations. \n\n1. The transformer  network is able to mix the information of all nodes in the subgraph, but it fails to model the structure of the subgraph. Therefore, it seems not a good choice to use transformer  to encode graphs. It is neccessary to explain how the transformer  model the graph strucutre. Why do not you not use GNN networks?\n2. When predicting the missing tail entity for (h,r,?), the methods need to sample and encode subgraphs for all the candidate tail entities. It seems  time-consuming. It is better to provide an analysis of  time complexity. \n", "summary_of_the_review": "This paper is generally well written and the solution is ok. If the authors can clarity my questions, I will re-evaluate the paper. ", "strength_and_weaknesses": "Weakness: \n\n1. The transformer  network is able to mix the information of all nodes in the subgraph, but it fails to model the structure of the subgraph. Therefore, it seems not a good choice to use transformer  to encode graphs. It is neccessary to explain how the transformer  model the graph strucutre. Why do not you not use GNN networks?\n2. When predicting the missing tail entity for (h,r,?), the methods need to sample and encode subgraphs for all the candidate tail entities. It seems  time-consuming. It is better to provide an analysis of  time complexity. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject