Clustering then Propagation: Select Better Anchors for Knowledge Graph Embedding
Authors: KE LIANG
Yue Liu
Hao Li
Lingyuan Meng
Suyuan Liu
Siwei Wang
sihang zhou
Xinwang Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Anchor-based Method
Clustering
Large Scale
Knowledge Graph Embedding
Primary area: learning_theory

Abstract: Traditional knowledge graph embedding (KGE) models map entities and relations to unique embedding vectors in a shallow lookup manner. As the scale of data becomes larger, this manner will raise unaffordable computational costs. Anchor-based strategies have been treated as effective ways to alleviate such efficiency problems by propagation on representative entities instead of the whole graph. However, most existing anchor-based KGE models select the anchors in a primitive manner, which limits their performance. To this end, we propose a novel anchor-based strategy for KGE, i.e., a relational clustering-based anchor selection strategy (RecPiece), where two characteristics are leveraged, i.e., (1) representative ability of the cluster centroids and (2) descriptive ability of relation types in KGs. Specifically, we first perform clustering over features of factual triplets instead of entities, where cluster number is naturally set as number of relation types since each fact can be characterized by its relation in KGs. Then, representative triplets are selected around the clustering centroids, further mapped into corresponding anchor entities. Extensive experiments on six datasets show that RecPiece achieves higher performances but comparable or even fewer parameters compared to previous anchor-based KGE models, indicating that our model can select better anchors in a more scalable way.

Clustering then Propagation: Select Better Anchors for Knowledge Graph Embedding
Ke Liang'
Yue Liu'
Hao Li
Lingyuan Meng' Suyuan Liu' Sihang Zhou' Xinwang Liu
Siwei Wang
National University of Defense Technology, Changsha, China Academy of Military Sciences, Beijing, China
Abstract
Traditional knowledge graph embedding (KGE) models map entities and relations unique embedding vectors in shallow lookup manner. As the scale of data becomes larger, this manner will raise unaffordable computational costs. Anchor- based strategies have been treated effective ways to alleviate such efficiency problems by propagation on representative entities instead of the whole graph. However, most existing mechr-bbsee KGE models select the anchors in a primitive manner, which limits their performance. To this end, we propose a novel anchor- based strategy for KGE, i.e., relational clustering -aased anchor selection strategy (RecPiece), where two characteristics are leveraged, i.e., (1) representative ability of the cluster centroids and (2) descriptive ability of relation types in KGs. Specifically, we first perform clustering over features of factual triplets instead of entities, where cluster number is naturally set as number of relation types since each fact can be characterized by its relation in KGs. Then, representative triplets are selected around the clustering centroids and further mapped into corresponding anchor entities. Extensive experiments on six datasets show that RecPiece achieves higher performances but comparable or even fewer parameters compared to previous anchor- based KGE models, indicating that our model can select better anchors in a more scalable way.
Introduction
Knowledge graphs (KGs) (35, such as Freebase [5] Wikidata 69 consist of a large number
of relational facts, such as Freebase (5), Wikidata 69
YAGO 51 and NELI [8] consist of
large number of relational facts, which are generally in the format of triplets, i.e., (head entity, relation, tail entity). Each triplet in KGs reveals specific connection between entities. To leverage
such informative knowledge to enhance the capacity of models different fields 775
24 84
and
applications 440 43 38 multiple knowledge graph embedding (KGE) models [59 60
48
82
32
31
35
36
80]
have been proposed these years.
However, traditional KGE models usually encode the entities, relations, and factual triplets in KGs shallowly. Assuming the dimension as traditional KGE models, such as RotatE [1], will map these elements in KGs into subspace R.V xd where N is the number of the target objects. Such a shallow lookup manner in these traditional KGE models results in a linear growth of memory consumption for storing the embedding matrix and incurs high computational costs 13] Thus, as the scale of data becomes larger and larger, top-level GPU or CPU clusters with more memory space are required for these traditional KGE models. For example, about 78M x 200d entity feature matrix and 58.1 GB GPU RAM 20 are needed for the best-performing model on
dataset 229
CCoressponiing author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Entity id:
Entily id: I
Entily id: 2
SO
Entity id: N
Shallow Knowledge Graph Embedding
Complexity Increases along with Entity Space Figure The problem of shallow knowledge graph embedding method.
To address such efficiency problems, there are three typical strategies integrated with KGE models, including quantification (37 53 7T). knowledge distillation [22 72 55 [85] and anchor-based sampling (20) [7) 30, (78). However, the first two types of strategies aim to realize more lightweight models for deployment by compressing the normal-size trained model. In other words, the standard KGE models still need to be trained on large datasets in advance. Compared to them, the anchor- sampling strategy, raised by [2], will be more efficient both training and deployment phrases, since it actually reduces the size of the entity set for propagation, i.e., from an intact set of all of the entities to an anchor set of some representative entities. Note that propagation means the feature aggregation procedure. Compared to quantification and knowledge distillation, anchor-based sampling can be easily composed with different KGE baselines to optimize the efficiency of these models.
30 78 select the anchors in a primitive manner, e.g., random selection, and manual selection of different centrality measurement strategies (20), etc. Thus, the anchor quality cannot be well guaranteed. which limits the performance of the models. In particular, the weights for each strategy on different datasets are usually determined according to grid searching, which is snoource-consumigg In addition, considering efficiency will generally bring performance loss compared to the selected shallow KGE backbones. Therefore, reducing performance loss while ensuring good efficiency is also a problem that our work expects to solve. More related works are discussed in Appendix A.2 due to the space limitation.
While, most of the existing anchor-based KGE models 120 70
Our work takes an attempt to design a more reasonable and accurate anchor selection strategy for better-quality knowledge embedding. During the investigation, two important characteristics come to our sights and are used in our RecPiece, including: (1) representative ability of the cluster centroids and (2) descriptive ability of relation types in KGs. Specifically, cluster centroids are proven as the most representative samples within corresponding clusters in various works [67| 52 26 46). Meanwhile, clustering will not introduce too many procedures via unsupervised learning techniques. which is proper to be adopted as the core mechanism for anchor selection in an efficient KGE model. Furthermore, typical clustering algorithms require two inputs, i.e., clustering features and cluster number. Both of them should be determined according to the characteristics of KGs, thus leading the clustering- based mechanism more suitable for the data type of KGs. As known to all, KGs focus more on relationships between entities compared to other graph types, so each factual triplet in KGs reveals relational knowledge. addition, triplets can be easily categorized into different clusters according to relation types in KGs. For example, (Mike, father of. Tom) and (John, father of. James) can both be characterized into same type, i.e., facts to reveal the "father of" relationship. The characteristic shows the descriptive ability of relation types in KGs. Inspired by it, we select features of factual triplets instead of entities as the clustering features, and the number of relation types is set as the cluster number. Note both of the information can be easily obtained as the attributes in any KGs.
To this end, we propose RecPiece, novel anchor-based KGE model with a relational clustering- anchor selection strategy. Specifically, we perform clustering over the features of the relational facts instead of entities, where the cluster number is naturally set to the number of relation types since each fact can be characterized by its relation in KGs. Then, the representative triplets are selected around the clustering centroids, which are further mapped into corresponding anchor entities. Extensive experiments are conducted on both link prediction and entity classification among RecPiece, shallow KGE models, and typical anchor-based KGE baseline, i.e., NodePiece, to demonstrate the promising capacity of our RecPiece from six aspects, i.e., superiority, effectiveness, scalability, efficiency, transferability, and sensitivity. In summary, the contributions are shown from three aspects below:
Problem.
We analyze the limitations of previous anchor-based KGE models, and point out two useful characteristics: (1) representative ability of the cluster centroids and (2) descriptive ability of relation types in KGs, which guide RecPiece to address the limitations.

---

Method. We design a novel anchor-based KGE model with a relational clustering- anchor selection strategy. In particular, we perform clustering on the features of factual triplet into [R] (the number of relation types) clusters, which can be easily determined as the attributes of any given KGs, thus leading to a more scalable and explainable efficient KGE model.
Extensive experiments show that RecPiece endow shallow KGE models to have better efficiency but without significant performance losses compared to other anchor-basee based KGE models on various downstream tasks, indicating that our model can select better anchors in more scalable way. In particular, RecPiece is not only 84x and 1 1.2x lightweight than shallow KGE baseline, i.e., AutoSF, and anchor- based KGE baseline, i.e,, NodePiece, but also makes 9.5% and 4.9% ranking performance improvements on MRR.
Experiment.
2 Related Work
This section summarizes the recent related works from three aspects: i.e., traditional knowledge graph embedding (KGE) model. parameter-efficiem model, and anchor-based strategy. Due to the space limitation, please refer to Appendix A.2 for details.
3 Method
The methodology of our RecPiece is illustrated in this section. More concretely, we first formulate the task and present the overall framework of RecPiece. Then, we further introduce the modules and procedures within RecPicce in detail, especially for the relational clustering-basee anchor selection procedure. At last, we provide a comprehensive discussion on the excellent attributes of RecPiece, which is enlightening for understanding our model. The framework of RecPiece is shown in Fig.
3.1 Prelinmary
Table I: Notation summary.
Task Formulation The knowledge graph is the di- rected relational graph, denoted as KG (E, R, 9). where E, R and G represent the set of entities (i.e, nodes), relations (i.e,, edge types) and fact triplets (i.e., edges), respectively. Similar to typical anchor- based baseline (20). RecPiece are more like plug- and-play auxiliary module, which can be easily ap- plied to any KGE model to reduce the space com- plexity of the adopted KGE backbone. Moreover, RecPiece is evaluated on different downstream tasks, i... link prediction and entity classification. Note that the focus of this work is not only on those rank- ing and classification metrics but also the efficiency. In other words, the main goal of our RecPiece is to achieve better performances different tasks with fewer or comparable parameters compared to the pre- vious anchor-based KGE models.
Notation
Explanation
E,R,9
set of entity, relation and triplet element of entity, relation, fact fact set for relation n; pretrained triplet encoder clustering algorithm candidate triplet selection mechanism triplet-entity mapping mechanisms
e.r.f
6:
p(-)
9(-)
(a(-)
(x(+)
[(-)
KGE models for feature propagation
|
quantity number
H,
feature matrix and vector
cluster centroid set ith cluster centroid distance function distance
dist(-)
T T
candidate triplct set and subset cluster
candidate anchor triplet distribution hased on relations number of anchors (typer-parantter
e
k
store the relational facts intuitively. Compared to other graph data, KG focuses more on the relationships between entities, and the knowledge is stored the factual triplets. Moreover, considering adopting the clustering algorithm for anchor selection, the cluster number can be easily fetched according to the number of relation types. Thus, we selected the features of triplets as the clustering features this paper.
Knowledge Graph Characteristic Knowledge graphs (KGs) [35 25
3.2 Overview Framework
Our RecPiece is novel anchor-based KGE model with relational clustering- based anchor selection strategy, which contains five procedures as shown in Fig. 2 including (a) feature preparation, (b) clustering over features of factual triplets, (c. candidate triplet selection, (c.2) triplet-entity mapping and (d) feature propagation. As anchors in RecPiece are selected based on the clustering based mechanism, need to generate the clustering features and determine the cluster number in advance. Thus, encoder p(-) is adopted for triplet feature preparation for clustering in (a). Then, the generated features are clustered into |R] clusters via g()) during (b). Later on, construct the anchor set via two procedures, (c.1) and (c.2). Finally, feature propagation happens on the anchors constrained with different task losses in (d). More details are described as follows, and notations refer to Tab.
3

---

(a) Feature Preparation
(b) Clustering over Features
R
p(-)
g(-)
(c.1) (c.I)dCadidatt Triplet Selection
&a (.)
Prediction
f(.)
Φ (-)
Entity Classification
(d) Feature Propagation
(c.2) Triplct-Entity Mapping
Figure 2: The illustration of our model, which is an anchor-based KGE model, termed RecPiece. by introducing a more explainable and scalable relational glustering-basee anchor selection strategy. Note that different factual triplets are coloured in different colours according to relation types, i.e. blue for 71, orange for r2, and green for T3. p(-) and f(-) are two encoders for feature preparation and propagation, respectively. Besides, g(.) is the adopted clustering algorithm with the cluster number set as the number of relation type [R], and anchor set construction contains two steps (...)) and (.)()) Note that the detailed description of the above modules is illustrated in Section and the notations are summarized in Tab. 1
3.3 Feature Preparation
Feature preparation aims to generate the features for clustering. During the procedure, the encoder p(-) takes the triplet set as input and outputs the corresponding feature matrix Hg E IR xl where the feature vector h, of triplet t; can be easily fetched from the corresponding row as Hgli.
Hg
p(G)
Specifically, p(-) contains the following steps: we first generate the embeddings of entities and relations with selected knowledge graph encoders. Then, we traverse all factual triplets in & and get the feature vector h,, for th triplet t, (ee) by summing up the normalized embeddings h of entities (he he,) and relation (h,). Finally, the triplet feature matrix H, is generated by concatenating all triplet embeddings together.
Hg
+ h In EG
(2)
3.4 Relational Clustering-based Anchor Selection As the core of RecPiece, novel relational clustering- anchor selection strategy is designed based on characteristics of both clustering centroids and knowledge graphs. Specifically, the selection strategy be separated into two parts, i.e., cluster centroid generation and anchor set construction.
3.4.1 Cluster Centroid Generation
We first perform clustering over the priorly generated triplet feature matrix, i.e., H. into | clusters, where cluster number set as the number of the relation types. The adopted clustering algorithm g(-)
output the embedding vector set of cluster centroids C (hey: Deg h R
C = g H, (R]
(3)

---

3,4.2 Anchor Set Construction
The cluster centroids are usually not the specific samples in datasets. Thus, we find representative samples anchors around them. To achieve the goal, procedures designed for anchor set construction, including candidate triplet selection (x(+) and triplet-entity mapping Φs(+). Concretely, the former procedure aims to select representative triplets for each relation type, while the latter procedure picks up the representative entities based on the triplets to constitute the final anchor set.
Candidate Triplet Selection. The factual triplets with the top m, closest distance the cluster centroid C, & C are selected as the candidate triplets. Specifically, cosine(-) function is used as the distance function dist(-) to measure the distance, which can also be substituted into other functions. such as euclidean(-). Then, we can get the distance set 'D for each cluster:
Dj dist (ctz (n)
I. Eg
(4)
m.
H
where tn E G denote the th factual triplet. and ha, represents the feature vector for centroid cy.
T
U
(E[1,]R]
(5)
U (E
arg top-mn D; t.
where the candidate triplet set T composed sets for different relation types. Triplets, whose embeddings are the top m, closest to he; E C, are selected the candidate triplets via arg top-m; Note that my is not a hyperparameter, and can be calculated when given the total anchor number
9 G
m, k
(6)
where r; represents the 2th relation type and | is the frequency distribution of triplets with relation ri. which can be easily obtained based on the attribute of the KGs. Taking the KG in Fig. as an example, the [[ [G2). Goll = [7. 7, 7] and rs, and if 1: = 6, [m,, m2, ma] = [2, 2|.
Triplet-Entity Mapping. The entity anchor set A is constructed from the selected triplet set T by randomly picking either the head or tail entity as the anchor entity corresponding to each triplet. The |A| < [T"), since will remove those identical entities.
A= (T*)))
3.5 Feature Propagation
The hashing and encoding procedures in (20] are leveraged for feature propagation. Specifically, the features of each entity e; are first hashed into a hash(e;) using two types of anchor information, i.e., discrete distances and relational contexts. Then, we leverage MLP as the encoder f(- to bootstrap the feature embeddings each entity based on the vectorized hashing features. Besides, according to the types of the downstream tasks, i.e., entity classification and link prediction, various loss functions adopted for training and optimization. In conclusion, our RecPiece can be integrated with different combinations of KGE backbones and loss functions toward different tasks and scenarios.
3.6 Attributes of RecPiece
In this section, we further discuss some attributes of the proposed RecPiece from various aspects shown below. (1) Random and manual anchor selection in previous anchor-based models is highly dependent on the human experience. Compared to them, ours is more reasonable and learnable according to the representative ability of the cluster centroids. (2) Our RecPiece is developed based on the characteristies of KGs. Specifically, we perform clustering on features of triplets instead of entities since the knowledge units KG are stored in triplets, which can also be easily characterized based on the relation type. (3) The hyper-parameter, ie., cluster number, for clustering algorithms can be determined according to the attributes in KGs in RecPiece as the number of relation types. Thus, anchor selection only contains one paramee,,,,,,er, i.e., anchor number, which is inevitable and
5

---

Table 2: Link prediction results F315k-237, WN18RR, CoDEX-L, and YAGO 3-10. % denotes KGE baseline, ie., NodePiece + RotatE.
the Hits @ 10 ratio regard to the
FB15k-237
WNISRR
Faarameter (M)
MRR
Hi
E0Rt
Parrmmeter
MRR
Hos
Rotath:
29.3
0.338
0,553
100
0.012
40.6
0,476
0.571
100
0012
NodePiece RotatE:
0.254
78.8
0.079
0.504
RecPiece RotatE
199
0.265
0.431
50.9
0.091
530
0.402
0.306
58.6
0.080
Improvement
4.3%
CoDEx-
YAGO
3-10
#laramatter (M)
MRR
Hits 10
E00k
Harameter
MRR
Hitse 10
RotatE: (500d)
77.0
0.258
0.387
100
0.033
123.0
0.495
0.670
190
0.004
(20d)
4.8
NodePleee + Rotatt
3.6
0.150
0,313
80.9
0.033
4.1
0.251
0.465
69.4
0.055
Improvement
16.7%
4.2%
3.2%
3,2%
24.5%
08
5.2%
3.7%
3.6%
5.5%
Table 3: Entity classification results on two subsets in WD50k.
WD50K (5% (abcled)
WDSOK (5% Inboled)
#Parameter (M)
ROC-AUIC
PRC-ALIC
Hard Ace
ROC-AIIC
PRC-ALC
Hard Ace
MLP COMPOCN Nodepise-CUMPPCCA
4.1
0,503
0.016
0.00)
0.510
0.017
0,002
4.4
0.836
0.280
0.176
0.834
0.265
0.161
0.75
0.516
RecPiece + COMPOCN
0.64
0,983
0.459
0.538
0.984
0.464
0,536
Improvement
14.7%
also needed by other anchor-based based methods. Besides, other models even need essource-consuminn grid-searching to get weights for different centrality measurement strategies on different KGs. (4)
Based on the above analyses, RecPiece is also more capable of being extended to various KGE models and applied to different KGs.
4 Experiment
Experiments are conducted to demonstrate the promising capacity of our RecPiece from five aspects, iee. superiority, effectiveness, scalability, efficiency, transferability, and sensitivity, by answering the following six questions.
Q1: Superority. Does RecPiece achieve better performance compared the previous anchor- based strategy, NodePiece, when integrated with different KGE models for different downstream tasks?
Q2: Effectiveness. Does the clustering strategy make a difference? Besides, how do the adopted components RecPiece influence the performance?
Q3: Efficiency. Will the RecPiece lead to a more parameter-efficient model? What is the performance of time and memory cost?
Q4: Scalability. How does our RecPiece perform on large-scale knowledge graph?
QS: Transferability. Will our RecPiece be effectively integrated with different KGE backbones?
Q6: Sensitivity. How does the performance influenced by RecPiece with different hyper- parameters?
4.1 Experiment Setting
Table 4: Dataset Statistic. "I.P" and "EC" denote link prediction and entity classification. "#" repre- sents the number.
Datasets. Six benchmark datasets are lever- aged to evaluate our RecPiece as same as previ-
ous works do [3
20
30 Specifically, FB15k-
237 [64), WNI8RR 18 CoDEx-L [54], and
Data
Task
#Ent.
#Rel.
#Fact
are used for link prediction. The entity classification is carried out two subsets (5% and 10% labeled) from WD50K (21], and the OGB WIKIKG 2 [23] 20 is the larger KGs for scalability analysis. The statistic details of the datasets are shown in Tab. 9
YAGO3-10
4
15k-237
L.P
14,505
237
310,079
WNISRR
40.559
11
92,583
CoDEx-L
77.951
69
612.437
YAGO3-10
LP
123,143
37
1,089,000
OGB WikikG
2,500,604
535
17,137,181
WD50K
EC
46.164
526
222,563
Implementation Details. All experiments are conducted on the server with 4-core Intel(R) Xeon(R)
Platinum 8358 CPUs 2.60GHZ, single 80 GB 1000 GPU and 64GB RAM with Py (49 libraries.

---

The p(-) for feature preparation is selected as pretrained NodePiece 63 the first few epochs. Besides, k-means 41 is selected as g(.) for clustering, and the cluster number is set as "#Rel." in Tab. for different datasets. For a fair comparison, we set anchor numbers & for each dataset as the same as (20), and 2-layer- MLP is adopted as f(-) feature propagation. In addition, we replace the default p(-), g(-) and
Table 5: Ablation study for different anchor se- lection strategies. "EP" and "RP" represent entity prediction and relation prediction. "NDC" and "PPR" are short for Node Degree Centrality and
Personalized PageRank.
Eniity Prediction
Relation Prediction
MRR
Hite 10
MRR
Hits@ 10
Random
0.249
0.417
0.878
0,971
NDC
0.250
0.418
0.877
0,970
dist(.) to pretrained GralL 633. BitechingK-
0.251
0.419
0.878
0,971
NodePiece
0.254
0.420
0.881
0,970
means [50] and euclidean(-) for robustness anal- ysis. Moreover, as for different tasks, RecPiece integrated with three KGE backbones. i.e., Ro-
RecPiece
0.265
0.431
0.884
0.975
tatE [61], ComPGCN [68] and AutoSF [79] to compare with thirteen KGE models, including (1)
link prediction: TransE (6) DisMult (77), ComplEX (65], PairRE (10), RotatE [61], TripleRE (78),
AutoSF (79), LRE + PairRE (I],, NodePiece + RotatE [20), and NodePiece + AutoSF (20]: (2) entity
classification: MLP, ComPGCN (88, and NodePiece + ComPGCN 20 More details are present in Appendix.
Evaluation Metrics. For link prediction, both MRR [16] and Hits@k [ are used the ranking metrics. Besides, ROC-AUC, PRC-AUC, AP, and Hard Accuracy are the evaluation metrics [23] for entity classification. To quantify the efficiency, we report the parameter number #P (M), memory cost (GB), running time (hours), and Effi. [3] Note that Effi. [3] is calculated by MRR/#P.
4.2 Main Performance (RQ1)
The performance comparison is carried between our RecPiece and the existing anchor-based KGE baseline, i.e., NodePiece, on two typical downstream tasks, i.e., link prediction and entity classification. It aims to answer Q1.
Results Report Tab. 2 shows that RecPiece can achieve better performance on link predic- tion, Le, average 6.5% fewer on parameter number and 3.5% and 2.5% improvements on MRR and Hits @ 10. In particular, the improve- ments are apparent on FB 15k-237 and CoDEX-L. Even though the performance improvements WN18RR are the smallest, it is still comparable. According to Tab. 3 RecPiece promising results on entity classification with about 14.7% fewer in parameter number and 3.3% and 4.4% boosts on PRC-AUC and Hard ACC, respectively.
Table 6:
Ablation study for whether pretrained based on language models. "KG-self" and "PLM" represent that the pretrained features generated on structure information and extra-textual informa- tion, respectively.
Model
MRR
Hits@ 10
also achieve
RotatE
0.338
0.533
NodePiece RotalE
0.254
0.420
RecPiece (KG-self)- RotatE
0.265
0.431
RecPiece (PLM) RotatE
0.262
0.425
Table 7: Ablation study for different clustering features. "triplet" and "entity" represent clustering the features of relational triplets and entities, respectively. All the results are for link prediction results on FB15k-237
Discussion Based on the above results, we can easily get the answer Q1 that our RecPiece can achieve better performances on both link prediction and entity classification with com- parable or even fewer parameters compared to the previous anchor-based strategy. NodePiece [20]. It further indicates less performance loss will be caused by RecPiece in more parameter- efficient manner. Although our RecPiece still
Model
MRR
Hits 10
RotatE
0.338
0.533
NodePicce RotatE
0.254
0.420
RocPicee (tripeet) + RotatE
0.265
0.431
RecPiece (entity) + Rotatl
0.259
0.424
raises the performance loss compared the shallow KGE baselines, i.e., RotatE and COMPGCN, it is an inevitable trade-off for considering efficiency (over 90% reduction on parameters) via anchor-based strategy, which also occurs on other anchor-based KGE models. Moreover, we notice that different performance improvements are made by our RecPiece in different datasets. It may suggest that our RecPiece can achieve better performance on those denser datasets for link prediction, thus leading to fewer improvements sparser WN INNR compared to denser FB F515k-237.
7

---

0.6
Kireens Krrectinn
0.4
0.45
0.45
W.D. distt)
0.3
0.3
MAR
14
HRrS11
H
Hlsg18
(b)
Figure 3: Ablation study of different components in RecPiece. (a), (b) and (c) show the impact the different pretrained triplet encoder p(-), clustering algorithm g(-), and distance function dist(.) for link prediction task on FB 1kk-237
4.3 Ablation Studies (RQ2)
In this section, different ablation studies are presented prove the effectiveness of RecPiece. Concretely, we first discuss the effectiveness of our anchor-selection strategy. Then, we further analyze the effectiveness of the important components in each step of RecPiece shown in Section 3.2 and Fig. 2
4.3.1 Anchor Selection Strategies
The anchors of better quality will definitely contribute to better performances. To prove that our relational clustering- based anchor selection strategy can effectively select better anchors, we compare it with four other strategies, i.e. Random Selection, Node Degree Centrality, Personalized PageRank, and NodePiece, FB 15k-3777 for link prediction, which contains two different settings, i.e., missing entity prediction and missing relation prediction. Tab. 5S) shows that our anchor selection strategy outperforms other strategies both two settings. particular, our strategy makes the 4.3% and 2.6% performance boost on MRR and Hits @ 10 metrics compared to the NodePiece-bel based model.
4.3.2 Feature Preparation In this section, we discuss the impact of different pretrain triplet encoders in the first step of RecPiece, i.e. feature preparation.
first adopt different structural information encoders p(.) ) KGs for pretraining. Fig. (a) shows that both GralL [63] and default RNP [20) can both achieve promising performance, but RNP is better than the GralL model. Besides, we also attempt to leverage the typical pretrained language model (PLM), i.e., BERT [27]. to prepare the pretrained features
the real textual meaning of different entities and relations. The results are shown in Tab. VII. It indicates that relying on structural information for pretraining reasonable that link prediction is indeed a task more related to network structures. However, the results also show the potential capacity of our RecPiece when leveraging the extra information.
KG link prediction is more promising than extra-textual meaning. It
our model, the pretrained procedure only used for feature preparation, which can be replaced as you want. No matter which feature preparation it is, the key idea of the paper will not be affected. Nevertheless, our RecPiece can all make improvements when leveraging different pretrained encoders.
4.3.3 Clustering As for the second step of RecPiece, we discuss and analyze the clustering features and the clustering methods.
Features
Clustering Features We also conduct experiments to verify that it is better to perform clustering relational triplet features. Table VI shows that although there still performance boosts when leveraging entity features, there are more apparent improvements in performances with triplet features. Thus, the results prove our idea, i.e., relational triplet features are more representative than entity features in KGs. Note that we try different cluster numbers for experiments on entity clustering and select the best results of them (with cluster number 10).
Clustering Methods We also try different clustering methods for anchor selection. i.e.. KMeans and BisectingKmeant Fig. (b) shows that both clustering methods can lead to promising performances. It further indicates that framework is effective with different clustering method choices, which demonstrates the generalizability of our model.
4.3.4 Anchor Selection
As for the anchor selection step, we further analyze the impact different distance functions to select the anchors that are closer to the clustering centroids. The experiments are carried out on two

---

25000
NodePPoce RecPiece
22735 22437
NodePicce RecPice 18:57
Rumming Time ([tuus)
Memory Cost (MB
20000
15
14:52
11.48
1057910397
10000
6145
2571 2505
3723 7087
2.97
2.38
3.97
CaDEx-L
Figure 4: Memory cost and running time comparison.
types of distance function, i.e,, Cosine function and Euclidean function. Fig. (c) demonstrates that RecPiece is robust to different distance functions, where Cosine(.) is better than Euclidean((). But no matter which strategies they are, the performances are higher than the performance without distance function. It further indicates the effectiveness and generalizability of our framework.
4.3.5 Discussion
We comprehensively present ablation studies from different aspects. Tracking all of the results of the experiments demonstrates the effectiveness of RecPiece, which is composed of the answer to Q2. Since the ablation study on feature propagation is to evalu- the effectiveness of different combinations of backbone KGE models, which is simi- lar to transferability analysis, the detailed discussion of this part is shown in Section 4.6.
4.4 Efficiency Analysis (RQ3)
This section presents and discusses the effi- ciency of RecPiece from three aspects, param-
Table 8: Link prediction results OGB WikiKG The best results are marked in Bold.
eter efficiency, memory efficiency, and time ef- ficiency. As shown in Tab. 2 and 'Tbb. 3 RecPiece can incredible save about 10x, 8x, 26x, 30x, and 7x parameters mark datasets, including FB 15k-237, WNISRR, CoDEx-L, YAGO 3-10 and WD50K compared to shallow KGE baselines, i.e., RotatE and COMPGCN. Compared NodePiece, the sults above prove that RecPiece can endow the KGE model to be more purameter-efficienge Specifically, performance boosts are made on both link prediction and entity classification with comparable or even fewer parameters. It
Model
#Paramns
MRR
TransE (500d)
1250M
0.426 #0.003
five bench-
DisMult (500d)
1250M
0.373 #0.005
RotatE (250d)
1250M
0.433 #0.002
ComplEX (250d)
1250M
0.503 #0.003
PairRE (200d)
500M
0,521 =0.003
PairEE+LRE (700d)
505M
0,584 + n/a
TripleRE
501M
0,579 #0.002
AutoSF
500M
0,546 #0.005
NodePiece AutoSF
6.9M
0,570 #0.003
RecPiece AutoSF
5.9M
0.598 #0.003
Improvement
14.5%
2.2%
further indicates less performance loss will be caused by RecPiece more efficient manner. Al- though we focus more on parameter efficiency, Fig. shows the recorded real GPU cost and running time are comparable and slightly less. Concretely, has better efficiency on real memory cost and running time, i.e., on average 14 MB and 0.89 hours reduction compared to NodePiece. It further indicates that RecPiece also has a good effect on memory efficiency and time efficiency.
In conclusion, the efficiency our RecPiece is promising (RQ3). We also admit that the pre-train paradigm and -ncher-based strategies will cost some resources, both in time and memory, which will be optimized in the future. Note that such redundancy is commonly seen other methods with similar techniques. Considering the performance gains, such limited cost is also acceptable, which should not influence the effectiveness of RecPiece.
4.5 Scalability Analysis (RQ4)
measure its scalability to larger KGs. According to Table 8 we observe that our RecPiece + AutoSF can outperform other KGE models. Specifically, the RecPiece + AutoSF model has only 5.9 M parameters. about 84x smaller than the most efficient shallow models, i.e., AutoSF. Meanwhile, it also about 1.2x lighter than anchor-based KGE baseline, i.e., NodePiece AutoSF 20), with the same quantity of anchors. Meanwhile, our RecPiece + AutoSF can even achieve better ranking performances compared the KGE model with best performance, i.e,, about 2.2% MRR improvement compared to PairRE + LRE (70). Moreover, the running time for NodePiece AutoSF and RecPiece + AutoSF is recorded
We compare RecPiece with nine state-of-the-arr KGE models on OGB WikiKG

---

0.6
033
49
0.08
190
30
PE1
Netric
[P.E]
(b)
(6)
Figure 5: Sensitive analysis in different settings. (a) and (b) reveals the influence of the models on feature preparation different epochs (i.e., [P.E.] represents the pretrained epoch number), where (a) is for entity classification on WD15K (5% labeled), and (b) is for link prediction on FB 1B15k-237. (c) indicates the influence of the total anchor number k. (c) is based on link prediction on FB15k-231
as 5.33 and 5.25 hours, which also comparable. We believe that the above results prove the better scalability of RecPiece with promising parameter reduction and better ranking performance.
4.6 Transferability Analysis (RQ5)
We answer Q5 in this section. Based the aforementioned experimental results cover two different downstream tasks, i.e., entity classification and link prediction, and three different types of shallow KGE backbones, i.e., RotatE, COMPGCN, and AutoSF, can reorganize and analyze the results from another view. First of all, all of the experimental results are promising. In particular, there are 5.2% MRR performance improvements made by RecPiece on YAGO 3-10 for link prediction when integrated with RotatE. Secondly, the improvements occur in all the situations when the backbone models are integrated with our RecPiece. Therefore, as the conclusion, our RecPiece is proven to be easily extended to different tasks and different KGE models plug-and-pla auxiliary mechanism, which shows great transferability of our RecPiece.
demonstrated that
4.7 Sensitivity Analysis (RQ6)
We measure the sensitivity of RecPiece from two aspects to answer Q6. including the analysis on (1) pre-trained epochs for the feature preparation model p(-). (2) anchor number k. The experiment results are shown in Fig. 5 In general, our RecPiece is insensitive to the hyperparameters, which demonstrates that RecPiece following two observations according to the aforementioned two aspects.
achieve stable performances. More specifically,
get the
(1) Fig. 5 (a) and (b) reveals that there usually exists pretrained epoch threshold pe for p(-), which indicates that the pretrained features are effective enough to be clustered for anchor selection after pe", e.g. around 800 epochs (in 4000 epochs) for entity classification WD15K (5% labeled) and 150 epochs (in 400 epochs) for link prediction on FB 15k-237.
(2) Fig. (b) shows that more anchors will benefit the performance tional KGE models propagate the features based on the whole graph, which is equivalent to the anchor set composed all entities. larger of anchors is closer the complete entity set so that less information will be abandoned, thus leading to better performance.
NodePiece. Tradi-
5 Conclusion
In this paper, we propose novel anchor-based KGE model with a relational clustering-based anchor selection strategy, RecPiece, where two characteristics are leveraged, i.e., (1) representative ability of the cluster centroids and (2) descriptive ability of relation types in KGs. Specifically, we perform clustering over the features of triplets instead of entities into RR| (number of relation types) clusters. Then, representative samples are selected around cluster centroids, which are further mapped into corresponding anchor entities. Extensive experiments show that RecPiece can endow shallow KGE models have fewer parameters without significant performance loss compared to other models, on various tasks, indicating that our model selects better anchors in more scalable way. In the future, we plan to optimize this preparation procedure via a aelf-vdaptive mechanism along with feature propagation for better practicability and adaptivity in the future.
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China (project no. 62325604, 62276271).
References
[1] Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Mikhail Galkin, Sahand Sharifzadeh, Asja Fischer, Volker Tresp, and Jens Lehmann. Bringing light into the dark: large-scale evaluation of knowledge graph embedding models under unified framework. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence, 44(12):8825-888 2021.
[2] Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Mikhail Galkin, Sahand Sharifzadeh, Asja Fischer, Volker Tresp, and Jens Lehmann. Bringing light into the dark: A large-scale evaluation of knowledge graph embedding models under a unified framework. IEEE Transactions on Patlern Analysis and Machine Intelligence, 44(12):8825-8844 2021.
[3] Youhui Bai, Cheng Li, Zhiqi Lin, Yufei Wu, Youshan Miao, Yunxin Liu, and Yinlong Xu. Efficient data loader for fast sampling-basee -baiid gnn training on large graphs. IEEE Transactions on Parallel and Distributed Systems, 32(10):2541-2556 2021.
[4] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135-146, 2017.
[5] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge. and Jamie Taylor. Freebase: A collaboratively created graph database for structuring human knowledge. Proceedings of the 2008 ACM SIGMOD International Conference Management Data, SIGMOD '08, page 1247-1250, New York, NY, USA, 2008, Association for Computing Machinery.
[6] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[7] Irene Cannistraci, Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, and Emanuele Rodolà. Bootstrapping parallel anchors for relative representations. arXiv preprint arXiv:2303.00721 2023.
[8] Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka, and Tom M. Mitchell. Toward an architecture for never-ending language learning. In Proceedings of the Twenty-Fourth AAAI Conference Artificial Intelligence, AAAI'10, page 1306-1313.
AAAI Press, 2010.
[9] Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low-dimensional hyperbolic knowledge graph embeddings. arXiv preprint arXiv:2005.00545, 2020.
[10] Linlin Chao, Jianshan He, Taifeng Wang, and Wei Chu. PairRE: Knowledge graph embeddings via paired relation vectors. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4360-4369, Online, August 2021. Association for Computational Linguistics.
[11] Linlin Chao, Taifeng Wang, and Wei Chu. Pie: a parameter and inference efficient solution for large scale knowledge graph embedding reasoning. arXiv preprint MXXi::2204.139577 2022.
[12] Linlin Chao, Taifeng Wang, and Wei Chu. Pie: parameter and inference efficient solution for large scale knowledge graph embedding reasoning. arXiv preprint arXiv:2204.13957, 2022. 2022.
[13] Mingyang Chen, Wen Zhang, Zhen Yao, Yushan Zhu, Yang Gao, Jeff Z. Pan, and Huajun Chen. aggostc representation learning for parameter-effiient knowledge graph embedding. In AAAI, 2023.
11

---

[14] Xiang Chen, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, Huajun Chen, and Ningyu Zhang. LightNER: A lightweight tuning paradigm for low-resource NER via pluggable prompting. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na, editors, Proceedings of the 29th International Conference on Computational Linguistics, pages 2374- 2387, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
[15] Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, and Huajun Chen. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion. In Proceedings of the 45th international ACM SIGIR conference research and development in information retrieval, pages 904-915, 2022.
[16] Nick Craswell. Mean Reciprocal Rank, pages 1703-1703. Springer US, Boston, MA, 2009.
[17] Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyper- spherical variational auto-encoders. arXiv preprint vrXiv:/804.0089) 2018.
[18] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Con- ference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 18/IAAI TAAAAI18/EAATI 18/EAAI 18. AAAI Press, 2018.
[19] Takuma Ebisu and Ryutaro Ichise. Generalized translation-based embedding of knowledge graph. IEEE Transactions on Knowledge and Data Engineering, 32(5):941 2020.
[20] Mikhail Galkin, Etienne Denis, Jiapeng Wu, and William I. Hamilton. Nodepiece: Compo- sitional and carameter-efficient representations of large knowledge graphs. arXiv preprint arXiv:2106. 12144. 2021.
[21] Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari, Ricardo Usbeck, and Jens Lehmann. Message passing for hyper-relational knowledge graphs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7346-7359, Online, November 2020. Association for Computational Linguistics.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 1503. 150..0253, 2015.
[23] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 13:22118-2221
2020.
[24] Feicheng Huang, Zhixin Li, Shengjia Chen, Canlong Zhang, and Huifang Ma. Image captioning with internal and external knowledge. In Proceedings of the 29th ACM International Conference on Information &amp; Knowledge Management, CIKM '20, page 535-544, New York, NY, USA, 2020. Association for Computing Machinery.
[25] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions neural networks and learning systems, 33(2):494-514, 2021.
[26] Zhao Kang, Zhiping Lin, Xiaofeng Zhu, and Wenbo Xu. Structured graph learning for scalable subspace clustering: From single view multiview. IEEE Transactions Cybernetics, 52(9):8976-8986 2021.
[27] Jacob Devlin Ming-We Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2, 2019.
(28) Thomas N Kipf and Max Welling. a7X1v:1611.07308, 2016.
Variational graph auto-encoders. arXiv preprint
12

---

[29] Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. Pytorch- biggrah: A large scale graph embedding system. Proceedings of Machine Learning and Systems, 1:120-131,200
[30] Hongzhu Li, Xiangrui Gao, Linhui Feng, Yafeng Deng. and Yuhui Yin. Stargraph: Knowledge representation learning based on incomplete two-hop subgraph.
[31] Qian Li. Cheng Ji, Shu Guo, Zhaoji Liang, Lihong Wang, and Jianxin Li. Multi-modal knowledge graph transformer framework for multi-mon multi-modal entity alignment. arXiv preprint arXiv:23 10.06365, 2023.
[32] Qian Li, Lixin Su, Jiashu Zhao, Long Xia, Hengyi Cai, Suqi Cheng, Hengzhu Tang, Junfeng Wang, and Dawei Yin. Text-video retrieval via multi-modal hypergraph networks. Pro- ceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 069-377,00 2024.
[33] Ke Liang, Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang, Xiangjun Dong, and Xinwang Liu. Knowledge graph contrastive learning based on -ymmmmrr--mmmmticcalll structure. IEEE Transactions on Knowledge and Data Engineering, 2023.
[34] Ke Liang, Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang. Xiangjun Dong, and Xinwang Liu. Knowledge graph contrastive learning based on relation-sy symmetrical structure. IEEE Transactions on Knowledge and Data Engineering, 36(1):226 238, 2024.
[35] Ke Liang. Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang Liu, Fuchun Sun, and Kunlun He. A survey of knowledge graph reasoning on graph types: Static, dynamic, and multi-modal. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1-20, 2024.
[36] Ke Liang, Lingyuan Meng. Sihang Zhou, Wenxuan Tu, Siwei Wang, Yue Liu, Meng Liu, Long Zhao, Xiangjun Dong, and Xinwang Liu. Mines: Message intercommunication for inductive relation reasoning over neighbor-enhanced subgraphs. Proceedings of the AAAI Conference Artificial Intelligence, 38(9); 1665510665--0653, Mar. 2024.
[37] Darryl D. Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML'16, page 2849-2858. JMLR.org, 2016.
[38] Chunyu Liu, Wei Wu, Siyu Wu, Lu Yuan, Rui Ding, Fuhui Zhou, and Qihui Wu. Social-enhanced explainable recommendation with knowledge graph. IEEE Transactions on Knowledge and Data Engineering, pages 1-14, 2023.
[39] Nelson Liu, Ananya Kumar, Percy Liang, and Robin Jia. Are sample-efficient nlp models more robust?, 2022.
[40] Yong Liu, Susen Yang, Yonghui Xu, Chunyan Miao, Min Wu, and Juyong Zhang. Con- textualized graph attention network for recommendation with item knowledge graph. IEEE Transactions on Knowledge and Data Engineering. 35(1):181-195, 2023.
[41] S. Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 28(2):129-137, 1982.
[42] Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. In International Conference on Learning Representations, 2024.
[43] Ziyu Lyu, Yue Wu, Junjie Lai, Min Yang, Chengming Li, and Wei Zhou. Knowledge enhanced graph neural networks for explainable recommendation. IEEE Transactions Knowledge and Data Engineering. 35(5):4954 2023.
[44] J. MacQueen. Some methods for classification and analysis of multivariate observations. 1967.
13

---

[45] Farzaneh Mahdisoltani, Joanna Asia Biega, and Fabian M. Suchanek. Yago3: A knowledge base from multilingual wikipedias. Conference on Innovative Data Systems Research, 2015.
[46] Dimity Miller, Niko Sunderhauf, Michael Milford, and Feras Dayoub. Class anchor clustering: A loss for distance-based open set recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3570-3578, 2021.
[47] George A Miller. WordNet: An electronic lexical database. MIT press, 1998.
[48] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2024.
[49] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Librury. Curran Associates Inc., Red Hook, NY, USA, 2019.
[50] Ruchi Patil and Amreen Khan. Bisecting k-means for clustering web log data. International Journal of Computer Applications, 116:36-41, 2015.
[51] Thomas Pellissier Tanon, Gerhard Weikum, and Fabian Suchanek. Yago 4: A reason-able knowledge base. In The Semantic Web: 17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31-June 4, 2020, Proceedings, page 583-596, Berlin, Heidelberg, 2020. Springer-Verlag.
[52] Qianyao Qiang. Bin Zhang, Fei Wang, and Feiping Nie. Fast multi-view discrete clustering with anchor graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9360-9367, 2021.
[53] Mrinmaya Sachan. Knowledge graph embedding compression. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2681-2691, Online, July 2020. Association for Computational Linguistics.
[54] Tara Safavi and Danai Koutra. CoDEx: A Comprehensive Knowledge Graph Completion Benchmark. In Proceedings the the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8328 8350, Online, November 2020. Association for Computa- tional Linguistics.
[55] Ahmed E Samy, Zekarias T. Kefato, and Sarunas Girdzijauskas. Graph2feat: Inductive link prediction via knowledge distillation. In Companion Proceedings of the ACM Web Conference 2023, pages 805-812, 2023.
[56] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer, 2018. 8.
[57] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. The Semantic Web: 15th International Conference, ESWC 2018 Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593-607. Springer, 2018
[58] Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 20/2 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149- 5152. IEEE, 2012.
[59] Lei Shi, Jia Luo, Chuangying Zhu, Feifei Kou, Gang Cheng, and Xia Liu. A survey cross- media search based on user intention understanding social networks. Information Fusion, 91:566-581, 2023.
[60] Lei Shi, Jiapeng Yang, Pengtao L.,, Yuan, Feifei Kou, Jia Luo, and Mingying Xu. Self-derived knowledge graph contrastive learning for recommendation. In ACM Multimedia 2024.
14

---

[61] Zhiqing Sun, Zhi-Hong Deng. Jian-You Yun Nie, and Jian Tang. Rotate: Knowledge graph embed- ding by relational rotation in complex space. arXiv preprint 1rX1v:1902.10197, 2019.
[62] Zhiqing Sun, Zhi-Hong Deng. Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph embed- ding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019.
[63] Komal K. Teru, Erienne Denis, and William L. Hamilton. Inductive relation prediction by subgraph reasoning. arXiv: Learning, 2020.
[64] Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509, Lisbon, Portugal, September 2015, Association for Computational Linguistics.
[65] Théo Trouillon, Johannes Welbl, Sehastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International conference machine learning, pages 2071-2080. PMLR, 2016.
[66] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071-2080 PMLR, 2016.
[67] Enmei Tu, Zihao Wang, Jie Yang, and Nikola Kasabov. Deep semi-supervised learning via dynamic anchor graph embedding latent space. Neural Networks, 146:350-360, 2022.
[68] Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition- multi-relational graph convolutional networks. In International Conference Learning Repre- sentations, 2020.
[69] Denny Vrandečić and Markus Krötzsch. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10)77--55, sep 2014.
[70] Baoxin Wang, Qingye Meng, Ziyue Wang, Dayong Wu, Wanxiang Che, Shijin Wang. Zhigang Chen, and Cong Liu. Interht: Knowledge graph embeddings by interaction between head and tail entities. arXiv preprint wxXiv2220.00499) 2022.
[71] Haoyu Wang, Yaqing Wang. Defu Lian, and Jing Gao. A lightweight knowledge graph embedding framework for efficient inference and storage. In Proceedings of the 30th ACM International Conference on Information &amp; Knowledge Management, CIKM '21, page 1909-1918, New York, NY, USA, 2021. Association for Computing Machinery.
[72] Kai Wang, Yu Liu, Qian Ma, and Quan Z Sheng. Mulde: Multi-teacher knowledge distillation for low-dimensional knowledge graph embeddings. In Proceedings of the Web Conference 202/, 1, pages 1716-1726, 2021.
[73] Meng Wang, Weijie Fu, Shijie Hao, Hengchang Liu, and Xindong Wu. Learning on big graph: Label inference and regularization with anchor hierarchy. IEEE Transactions on Knowledge and Data Engineering, 29(5): 101-1114,, 2017.
[74] Siwei Wang, Xinwang Liu, Suyuan Liu, Jiaqi Jin, Wenxuan Tu, Xinzhong Zhu, and En Zhu. Align then fusion: Generalized large-scale multi-view clustering with anchor matching corre- spondences. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[75] Wenhan Xiong. Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. arXiv preprint 2019.
[76] Lei Xu, Zhen Yu He, Kai Wang, Chang-Dong Wang, and Shu-Qiang Huang. Explicit message- passing heterogeneous graph neural network. IEEE Transactions on Knowledge and Data Engineering,
2023.
[77] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6577 2014.
15

---

[78] Long Yu, Zhicong Luo, Huanyong Liu, Deng Lin, Hongzhu Li. and Yafeng Deng. Triplere: Knowledge graph embeddings via tripled relation vectors, 2022.
[79] Yongqi Zhang, Quanming Yao, and James T Kwok. Bilinear scoring function search for knowledge graph learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[80] Yongqi Zhang, Zhanke Zhou, Quanming Yao. Xiaowen Chu, and Bo Han. Adaprop: Learning adaptive propagation for graph neural network based knowledge graph reasoning. In SIGKDD, 2023.
[81] Zhanke Zhou, Yongqi Zhang, Jiangchao Yao, Quanming Yao, and Bo Han. Less is more: One-shot subgraph reasoning large-scale knowledge graphs. In ICLR, 2024.
[82] Zhanke Zhou, Chenyu Zhou, Xuan Li, Jiangchao Yao, Quanming Yao, and Bo Han. On strengthening and defending graph reconstruction attack with markov chain approximation. In International Conference on Machine Learning. 2023.
[83] Man Zhu, Weixin Wu, Jeff Z. Pan, Jingyu Han, Pengfei Huang, and Qian Liu. Predicate invention based rdf data compression. In R Ichise, F Lecue, T Kawamura, D Zhao, S Muggleton, and K Kozaki, editors. Semantic Technology 8th Joint International Conference, JIST 2018, Proceedings, Lecture Notes in Computer Science (including subseries Lecture Notes in Artifi- cial Intelligence and Lecture Notes in Bioinformatics). pages 153-161, Germany, November 2018. Springer Verlag. 8th Joint International Semantic Technology Conference, JIST 2018 Conference date: 26-1 1-2018 Through 28- 1-2018.
[84] Xiangru Zhu, Zhixu Li, Xiaodan Wang, Xueyao Jiang, Penglei Sun, Xuwu Wang, Yanghua Xiao, and Nicholas Jing Yuan. Multi-modal knowledge graph construction and application: A survey. IEEE Transactions on Knowledge and Data Engineering, pages 1-20, 2022.
[85] Yushan Zhu, Wen Zhang, Mingyang Chen, Hui Chen, Xu Cheng, Wei Zhang, and Huajun Chen. Dualde: Dually distilling knowledge graph embedding for faster and cheaper reasoning. In Proceedings of the Fifteenth ACM International Conference Web Search and Data Mining, pages 1516-1524, 2022.
[86] Zhaocheng Zhu, Xinyu Yuan, Michael Galkin, Louis-Pasces Xhonneux, Ming Zhang, Maxime Gazeau, and Jian Tang. A net: A scalable path-based reasoning approach for knowledge graphs. Advances in Neural Information Processing Systems, 36, 2024.
16

---

A Appendix
A.1 Limitations and Future Works
Although proven effective and the idea is novel for large-scale KGE models, this paper only takes a primitive attempt to select better anchors via clustering. Specifically, our feature preparation is independent which requires auxiliary models. In this manner, the quality of anchors will be influenced the adopted model is not well-trained. Thus, we plan to optimize this preparation procedure via a self-adaptive mechanism along with feature propagation for better practicability and adaptivity in the future.
A.2 Related Work
This section summarizes the recent related works from three aspects: i.e., traditional knowledge graph embedding (KGE) model, parameter-efficient model, and anchor-based strategy. The discussion on typical related works provides the necessary background knowledge for our novel anchor-based paraneter-effiiient KGE model, which assists readers to better understand our work.
Traditional Knowledge Graph Embedding Model. Traditional knowledge graph embedding (KGE) models aim to map entities and relations into low-dimensional representations [25 2 33
while preserving the semantic information underlying the original KG. In general, it can be roughly
divided into three types, i.e., translational models 66
62 [9], tensor factorization models [57 66].
and graph neural network based models [28] 57
17
68 [6]. For example, RotatE [6T] proposes a
rotation- based translational method with complex- complex-valued embeddings to better infer the symmetry, anti-symmetry, inversion, and composition facts, which are widely used. As an improved model for RGCN [56.. COMPGCN [68] also jointly learns the representations with various entity-relation composition operations. Moreover, AutoSF [79] proposes an adaptive manner for KGE scoring function searching. which gains promising performance on large-scale scenarios. The above three typical KGE baselines constitute the backbone models this paper following previous works (20 10]. However, few of these traditional KGE models do consider the efficiency of parameters (20) and data compression [83] As the scale of data becomes larger, the shallow lookup in traditional KGE models will result in linear growth of memory consumption for storing larger embedding matrices and incurs high computational costs [20]. which limits their scalability to real-world scenarios. Our RecPiece is designed for more parameter-efficient representation learning on knowledge graphs.
Parameter-Effichhe Model. Different strategies have been leveraged for more efficient KGE models recent years, which can be roughly categorized into three types, i.e., quantification 37 53)
knowledge distillation 72 85,, and anchor- sampling [39.3] As for quantification-based methods, TS-CL (53) reduces the dimensions by learning discrete entity representations via quantiza- tion. Besides, LightKG 771 designs the residual module to induce diversity among codebooks and performs the dynamic negative sampling using quantization technology. Compared to them, MulDE (72). DualDE 185), and Graph2Feat [55] are all distillation-basee methods, which are developed based on the teacher-student frameworks. Specifically, MulDE 722 leverages multiple KGE models as teacher models to extract student model with lower dimension for space consumption reduction. However, the teacher model in DualDE [85] is more suitable for the student model to obtain better distillation results by considering the dual influence between them. Besides, Graph2Feat (55) extends the distillation procedure for inductive setting. However, the above two types of strategies aim to real- ize more lightweight models for deployment by compressing the trained model in normal size, which means that the standard KGF models still need to be trained on large datasets in advance. Compared to them, the anchor- mased- models (20 70 30 78) will be more efficient in both training and deployment since they actually reduce the size of the entity set for propagation. Moreover, anchor-based sampling can be easily composed with different KGE baselines to optimize the efficiency of these models.
Beyond the above categories, ATTH (9] utilizes hyperbolic space and geometric transformations to learn the improved www-dimnsiionll representations, and PIE [12 reduces the space consumption for entity representations via discrete code vectors and tensor decomposition. More recently, EARL 13 does not learn one vector for each entity but by learning only the embedding of small number of entities, encoding the distinguishing information from their connected relationships, k nearest reserved entities, and multi-hop neighbors, which translates the distinguishing information into entity embedding. **Net (86) proposes path-based propagation strategics. Meanwhile, AdaProp [80] and
17

---

Table 9: Dataset Statistic. "LP" and "EC" denote link prediction and entity classification. "#" represents the number.
Dalawets
Tasks
PEminis
*Rehiionsip
#Edges
STram
AValnbation
4Tes
2B13K-237
LP
14,305
237
310,079
272,115
17,526
20,135
WNTARR
1.P
40,559
11
92,583
86,835
2824
2924
CoDEx-Large
1.P
77,951
612,337
551,193
YAGO 3-10
LP
123.143
37
1,089,000 0,00000
1,079,010
4978
1,982
OGB WikiKG 2
1.
2,500,604
535
17,157,181
16,109,188
429,456
598,543
WD20K
BCC
46,164
526
222,563
4600N)
4600N)
One-shot subgraph method [81] propose novel subgraph- efficient methods, which are also an emerging direction these years.
In this work, we focus more on developing better and characteristic-adpphee anchor-based sampling strategy. Unlike other independent parameter-efficient methods, although the anchor-based methods may cause performance deduction, their alug-and-play attributes are better than other parameter- efficient methods. It has excellent potential to be easily integrated with any other KGE models, including hyperbolic embedding methods, such as ATTN (9]. Besides, among different types of efficiency of KGE models, such parameter efficiency, time efficiency, and memory efficiency, we are concentrating more on the parameter efficiency of traditional KGE models instead of targeting time and memory complexity.
Anchor-based Strategy. Anchor based strategies, as effective sampling techniques, are widely used in different fields [52) [73, [74, 667 [7] [7] to solve efficiency problems these years. The main idea of these strategies is select the representative sample to represent group of data so that the whole dataset is not required for the learning procedure, which will improve the scalability and efficiency of the original model. Inspired by Subword- poweredd algorithms
4 58
NLP, NodePiece (0 is
proposed, which is the most representative machor-basedd KGE model. Many KGE models come out based on it, such as TripleRE (78). Besides, StarGraph (30) and DigPiece 770 are proposed by adding extra neighbour constraints. Although these models all achieve promising performance, the anchors in these methods are still selected primitive manner, e.g., random selection, and manual selection of different centrality measurement strategies, including Node Degree Centrality, Personalized PageRank, etc. Since these selection strategies are neither reasonable nor developed according the characteristics of the data type of KGs, the quality of the anchors cannot be well guaranteed, which limits the performance of the models. In particular, the weight for each strategy on different datasets is determined according grid searching, which is time-consuming.
A.3 Experiment Setting
Experiment settings are introduced from three aspects, i.e., datasets, implementation details, and evaluation metrics.
A.3.1 Datasets
Six benchmark datasets are leveraged to evaluate our RecPiece as same as previous works do [13 20 30). Specifically. FB15k-237 664), WNISRR [18), CoDEx-L [54], and YAGO3-10 [45] are used for link prediction. Among them, FB15k-237 64 contains 237 relations, which is derived from Freebase [. and inverse relations are deleted to avoid the leaking problem compared to FB15k. Similar to it, WNI 8RR [8] derived from WordNet 47 without inverse relations. FB15k-237 can be considered relation-rich graphs, while WN18RR is sparse graph with few relation types. Besides, YAGO3-10 10 [45] consists of entities that have a minimum of 10 relations each which are extracted from YAGO3. Different from the above three typical traditional datasets, CoDEx (54) contains more diverse and interpretable content and is more difficult to evaluate. In this work, we used the largest subset of it. The entity classification is carried out two subsets (5% and 10% labeled) from WD50K (2T]. Furthermore, we leverage OGB WIKIKG 2 23 20) a large-scale KG, for scalability analysis. The statistical details of the datasets are shown in Tab.
A.3.2 Implementation Details
All experiments are conducted on the server with 4-core Intel(R) Xeon(R) Platinum 8358 CPUs @ 2.60GHZ, single 80 GB A100 GPU, and 64GB RAM with PyTorch [49 libraries. The p(-)
18

---

for feature preparation is selected as pretrained NodePiece [63] in the first few epochs. Besides. k-means 44 is selected as g(.) for clustering. and the cluster number is set "#Rel." in Tab.
for different datasets. Besides, a 2-layer-ML.P is adopted as f(()) feature propagation. In addition, BitechingKmeans
we replace the default pl-), g()) and d.st(.)) to pretrained Grall. 63
[50] and
euclidean(.) for ablation analysis.
Moreover, as for different tasks, RecPiece is integrated with three KGE backbones, i.e. RotatE
[T]. COMPGCN [68]. and AutoSF [79] to compare with thirteen KGE models, including (1) link prediction:
TransE [6, DisMult (77]. ComplEX 65]. PairRE [10], RotatE [1], TripleRE [78].
AutoSF 779.. LRE+ + PairRE [II], NodePiece + RotatE 20).. and NodePiece + AutoSF 20] (2) entity
classification: MLP, COMPGCN [68] and NodePiece COMPGCN [2).. Note that the anchor
selection in our RecPiece is relational clustering sesering-bbsedd anchor selection strategy. Compared to the manual selection strategy (composed of 20% Random Selection, 40% Personalized PageRank, and 40% Node Degree Centrality) in NodePiece, ours is more explainable and scalable. Here we present some descriptions of the used models as follows.
RotatE defines each relation as a rotation from the source entity to the target entity in the complex vector space, enabling it to model and infer various relation patterns, including symmetry, antisymmetry, inversion, and composition relations.
COMPGCN encodes entities and relations jointly by using various composition operators from KGE techniques, addressing the issue of peeeeaarameteiiaiionn in GCNs.
AutoSF proposed an algorithm that can automatically design and discover optimal scoring functions of the KGE model. Through a progressive greedy search algorithm, AutoSF can design promising KGE scoring functions effectively from vast search space.
TransE is a translation- based KGE model that aims to model inversion and composition relations. Inspired by the translation invariance in the word2vec model, TransE tries to make h + rest, where h, and represent the head entity, relation, and tail entity in triplet, respectively.
DistMult assumes that all relations in the KG are symmetric and represent them as block- diagonal matrices. Such a relation representation mechanism, combined with simple dot product operations, improves the efficiency of triplet evaluation.
ComplEx uses complex vectors instead of real vectors to represent the embeddings of entities and relations, which allows the model to distinguish between symmetric and asymmetric relations.
PairRE uses paired vectors to represent each relation, allowing the margin in the loss function to adjust adaptively. Thus, PairRE can express more complex relations, such as sub-relations.
TripleRE combines projection and translation operations. Specifically, the representation vectors of the head and tail entities are first projected and then translated to obtain the relation represen- tation. This method enriches the expression of relations, enabling the model to handle complex relations.
NodePiece proposed an efficient and plug-and-play node selection mechanism for KGE models. Specifically, NodePiece is inspired by WordPiece from the field of natural language processing, which is able to represent large-scale knowledge graphs using only fewer entity embeddings while also enhancing the generalization performance of the model at the same time.
LRE is high-efficienoe method that utilizes tensor decomposition to enhance the parameter efficiency of KGE models. Specifically, rather than decomposing the observed 3D tensor directly, LRE decomposes the entity embedding matrix to low-rank matrices.
A.3.3 Evaluation Metrics
Two types of evaluation metrics are adopted for two different downstream tasks. For link prediction, both mean reciprocal rank (MRR) [66 and @its@k @k [ are used as the ranking metrics, where k E {1, 3, 10). Besides, ROC-AUC, PRC-AUC, AP, and Hard Accuracy are the evaluation metrics (23 for entity classification. To quantify the efficiency, we report the parameter number #P (M), memory cost (GB), running time (hours), and Effi. [3], which is the metric recently proposed by [ to evaluate the efficiency of the KGE models.
MRR
Effi
(8)
# Parameter
Note that higher Effi. it is, the more efficient the model performs.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We present the contributions from three aspects, i.e., problem, method, and experiment.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss in our final section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental details are carefully described. Besides, the authors mention that the source code will be released after the double-bline review.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification:
The authors mention that the source code will be released after the double-blind
review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cmm ec/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http///ooSSuuiiss..eoeeee
//nips //nips.cc/pil cc/public/guu
htt.SSbbbiiiiooooiiiiiooooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of experimental settings are carefully described in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
Six aspects of experiments are conducted to evaluate the proposed model.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details of experimental settings are carefully described in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The authors preserve anonymity and obey the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper targets on the large-scale problem in knowledge graphs. By solving it, existing representation learning models can be applied to more general and realistic scenarios, such as retrieval on society relationship graph, etc.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of the backbone models and datasets are cited with their references.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "kJ9go6XLZ4", "reviewer": "Reviewer_z3Fe", "summary": "This paper proposes RecPiece, a novel anchor-based knowledge graph embedding (KGE) model that selects representative anchors via a relational clustering-based strategy. Specifically, RecPiece performs clustering over features of factual triplets instead of entities to generate cluster centroids, and the cluster number is set as the number of relation types. Representative triplets are then selected around the centroids and mapped to corresponding anchor entities. Extensive experiments on link prediction and entity classification show RecPiece achieves better performance with comparable or fewer parameters than previous anchor-based KGE models like NodePiece, demonstrating its ability to select better anchors in a more scalable way.", "questions": "1. How does the choice of the clustering algorithm affect RecPiece's performance? Will more advanced clustering methods bring further improvements?\n\n2. Besides the structure information, is it possible to make use multi-modal information during feature preparation?\n\n3. Is it possible to make use of LLM to enhance RecPiece in the future?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed relational clustering-based anchor selection strategy leverages the representative ability of cluster centroids and descriptive ability of relation types in knowledge graphs. This is a novel and reasonable approach compared to previous random or manual anchor selection.\n\n2. RecPiece is developed based on the characteristics of knowledge graphs by clustering on triplet features and using the number of relation types as cluster number. This makes the method more suitable for the data type of knowledge graphs.\n\n3. Extensive experiments on link prediction and entity classification across multiple datasets demonstrate the superiority, effectiveness, efficiency, scalability and transferability of RecPiece over baselines. For example, RecPiece achieves 3.5% and 2.5% improvements on MRR and Hits@10 for link prediction with 6.5% fewer parameters compared to NodePiece.\n\n4. RecPiece provides a simple yet effective plug-and-play module that can be easily integrated with various KGE models to reduce their space complexity and improve efficiency without incurring significant performance loss.", "weaknesses": "1. While RecPiece shows promising results, it still has some performance loss compared to the non-anchor based KGE models on some datasets. Though such trade-off between efficiency and effectiveness is expected, it would be better to provide some theoretical analysis on the potential causes of the performance gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions! We respond to your questions carefully one by one carefully, and we hope our responses can address your concerns.\n\n**RQ1. More Discussion on Trade-off between Efficiency and Effectiveness.**\n\nThanks, and we will respond from two aspects.\n\n**(1)**\tThe anchor-based KGE methods aim to narrow the view scope from the entire graph to the critical entities, thereby improving the efficiency of the model. As less information is utilized via this strategy, performance loss is usually inevitable, which also occurs on our RecPiece, a typical anchor-based KGE model. However, compared to previous anchor-based models, e.g., NodePiece, RecPiece can reduce performance loss and improve efficiency. In other words, RecPiece can achieve a promising performance with a better trade-off between efficiency and effectiveness. \n\n**(2)**\tFurthermore, we also agree with you and provide a theoretical analysis to illustrate two attributes, i.e., (i) the performance loss of the anchor-based KGE methods is inevitable, and (ii) the samples selected by RecPiece are more representative than random selection.\n\n**(i)** As mentioned in (1), anchor-based KGE methods actually try to select a proportion of the dataset to represent the whole dataset. However, the performance loss is inevitable with fewer samples, which is proven in many classic researches [1][2]. Here, we demonstrate it by relying on Theorem 1 in [1].\n\n$\\textbf{Theorem 1.}$ Assume $0<\\epsilon\\leq\\frac{1}{8}$, $0<\\delta\\leq\\frac{1}{100}$ and the  $VCdim(C)\\geq2$. The any $(\\epsilon, \\delta)$-learning algorithm A for C must use sample size \n\n$$m_A(\\epsilon, \\delta)\\geq\\frac{VCdim(C)-1}{32\\epsilon}=\\Omega(\\frac{ VCdim(C)}{\\epsilon}) (1) $$\n\nNote that the Vapnik-Chervonenkis dimension [3] of C is denoted as VCdim(C), which is the cardinality of the largest $W\\subseteq C$ such that $W$ is shattered by $C$. The $\\epsilon$ is the expected error.\n\nBased on the (1) above, we can get the following inequality (2)\n\n$$\\epsilon \\geq\\frac{VCdim(C)-1}{32 m_A(\\epsilon, \\delta)} (2),$$\n\nwhere we can easily get that when $VCdim(C)\\geq2$, the expected error $\\epsilon$ is lower bounded by $\\frac{VCdim(C)-1}{32m_A(\\epsilon, \\delta)}$. As the VCdim(C) in our case is determined as a constant once the learning algorithm $A$ is given, the lower bound of the expected error has negative correlations with the number of the samples $m_A$. Thus, fewer samples will lead to a larger lower bound, which will further result in worse performance.\n\n**(ii)** Assume the size of the original datasets is $N$, and the number of the categories is $k$, the more representative sampled dataset should at least follow one characteristic that there exists at least one sample belonging to each category. To demonstrate that our RecPiece makes it possible to select more representative samples compared to random selection, we can calculate the probability of both strategies for the abovementioned characteristics. Concretely, if the size of the sampled dataset is $m$ ($m\\geq k$), there are $C_N^m$ combinations for random selection, and the probability for it is $\\frac{1}{C_N^m}\\ll 1=\\operatorname{Prob(RecPiece)}$. It shows that the sampled sub-datasets generated from our strategy will contain all kinds of knowledge in the original datasets, which is more likely to lead to less performance loss.\n\n**RQ2. Influences of Clustering Algorithms.** \n\nThanks, and we will respond from three aspects.\n\n**(1)** The performance of anchor-based KGE methods highly corresponds to the quality of the anchors. Inspired by the representative capacity of the clustering centroid, our RecPiece utilizes clustering for anchor selection. If different clustering algorithms can all find the same or similar clustering centroids, the clustering algorithms will have less effect on RecPiece performance as the selected anchors may also be similar. Otherwise, different clustering algorithms will affect the RecPiece’s performances by affecting the selection of the anchor sets.\n\n**(2)** In our work, we focus more on introducing the idea of leveraging clustering for anchor selections in anchor-based KGE methods and emphasize the importance and advantages of performing clustering on relational facts rather than entities. Therefore, we use the most commonly used but effective clustering algorithm, $k-$means, to evaluate our idea. \n\n**(3)** Moreover, we discuss the influences of different clustering algorithms in Sec. 4.2.3 and Fig. 2 (b). Our paper provides more details.\n\n**RQ3. Utilization of Multi-modal Attributes.** \n\nThanks, and we will respond from two aspects.\n\n**(1)**\tSure, it is definitely possible to utilize other multi-modal attributes, and we have already considered it. Specifically, the performance comparison and discussion are present between different prepared features from both structural information and textual information in Sec. 4.2.2 and Tab. 4 of our paper. \n\n**(2)**\tWe further prepare the clustering features from visual information. Specifically, we make use of the multi-modal version of the FB15k-237 datasets from [4]. Then, we further leverage ViT [5] to generate the visual features of each entity. According to the Tab.1 in the attached PDF, we can also see that our RecPiece can achieve better performance with the visual attributes, which further demonstrates the scalability and effectiveness of our method.\n\n\n**RQ4. Corporation with our RecPiece and LLMs.** \n\nThanks! Due to the space limitation, please refer to RQ4 for Reviewer GCgr.\n\n**References.**\n\n[1] A general lower bound on the number of examples needed for learning\n\n[2] An introduction to computational learning theory\n\n[3] Learnability and the Vapnik-Chervonenkis dimension\n\n[4] A survey of knowledge graph reasoning on graph types: Static, dynamic, and multi-modal\n\n[5] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jgvmzHsCAh", "reviewer": "Reviewer_GCgr", "summary": "To address the computational inefficiencies of conventional knowledge graph embedding models, the authors of this study suggest RecPiece, an anchor selection technique based on relational clustering. RecPiece selects more efficient anchor entities by using the descriptive power of relation types and the representative ability of cluster centroids, which improves efficiency and scalability over earlier anchor-based methods.", "questions": "Several GNN methods work on the efficiency and scalability problem of KGs, such as AdaProp (KDD'23), AStarNet (NeurIPS'23), and One-shot-subgraph (ICLR'24). I suggest the paper discuss these recent works.\n\nAs large language models (LLMs) are widely studied, I recommend that the author discuss more about using LLMs for efficient KGE in future works. \n\nPlease also refer to the above weaknesses.", "limitations": "NA.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall writing of the paper is clear and well-structured. Clear figures are provided to help understand the proposed method, and the notations used are generally clear.\n\nThe efficiency and scalability problem investigated is important to the graph community. The paper is about a practical problem for graph-based scenarios. \n\nThe proposed method is simple yet effective. Specifically, it is a novel anchor-based method for knowledge graph representation that leverages clustering centroids and relation types. Compared to primitive anchor selection strategies, there is a great improvement as RecPiece can provide more representative and descriptive anchors for better performance on six datasets.\n\nThe experiments cover several datasets, with link prediction tasks, entity classification tasks, and relation prediction tasks. The scope of the experiments is extensive. Several ablation studies and empirical analyses are provided.", "weaknesses": "Only one embedding model, RotatE, is considered in experiments on datasets FB15k-237, WN18RR, CoDEx-L, and YAGO 3-10 (see Table 1). Other embedding models, such as TransE, DistMult, and ComplEx, should also be included here.\n\nThe writing of the paper can be improved. Some long paragraphs can be split into shorter ones. Besides, there are some repeated references, such as [1] and [2], [11] and [12], [51] and [52]. It would be better to check these references and unify the formats.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions! We respond to your questions carefully one by one carefully, and we hope our responses can address your concerns.\n\n**RQ1. More Compared KGE Backbones.** \n\nThanks for your suggestions, and we will answer your questions from three aspects as follows.\n\n**(1)**\tActually, RecPiece utilizes three kinds of KGE backbones instead of one model, including RotatE, CompGCN, and AutoSF, which reviewers vJzB and 89jh also mentioned.\n\n**(2)**\tFor a fair comparison, we follow the experiment settings in NodePiece [1]. Different KGE backbones are adopted for different downstream tasks. For example, RotatE is the KGE backbone for performance evaluation on link prediction on normal KG benchmarks and we also integrate our RecPiece with only one KGE backbone, i.e., RotatE, as same as NodePiece.\n\n**(3)**\tAs for TransE, DistMult and ComplEx, they are actually compared in the scalability analysis of Sec. 4.4 and Tab. 6 of the manuscript (Refer Tab. 2 in the attached PDF).\n\n**RQ2. Writing Issues**\n\nThanks for your suggestions. I will reorganize our paper into shorter paragraphs, especially for the second and third paragraphs of the introduction and related works. Besides, we will double-check our paper to revise typo mistakes, reference redundancy and unify the reference formats.\n\n**RQ3. Discussion on Advanced Models**\n\nThanks for your suggestions. Those are indeed advanced GNN methods that work on the efficiency and scalability problem of KGs, and we will discuss them in our revised version. Concretely, the discussion on them, including AdaProp [2], AStarNet [3], and One-shot-subgraph [4], will be presented in the second part of the related work section, i.e., Parameter-Efficient Model, because they are not anchor-based KGE models.\n\n**RQ4. Corporation with our RecPiece and LLMs** \n\nThanks for your valuable suggestions. We will answer it from two aspects, and add the discussion in our revised paper as the future works.\n\n**(1) Enhancing RecPiece with LLMs.** As shown in Fig. 1 of our manuscript, there are four steps in our RecPiece. As the main goal for our RecPiece is to achieve a better trade-off between efficiency and effectiveness, integrating LLMs into our framework, especially for the embedding models, will raise the parameter redundancy except for the first step, i.e., feature preparation. Concretely, there are two potential ways to utilize LLMs. On the one hand, we can utilize LLMs for textual information encoding, as shown in Fig. 1 (a) in the attached PDF. Actually, we have already tried to leverage PLMs for it in Sec. 4.2.2 and Tab. 4. Compared to PLMs, LLMs are of better expressive capacity, which will lead to better representation. On the other hand, when applying our RecPiece to real-world scenarios, we can use LLM-based agents for real-time information retrieval and collection, as shown in Fig. 1 (b) in the attached PDF. It saves the expensive labor cost for information collection before feature preparation.\n\n**(2) Enhancing LLMs with RecPiece.** Our RecPiece can be treated as an efficient feature encoding method, which is an option for utilizing large-scale KGs in our real-world applications. With the generated embeddings for the structural knowledge, LLMs can be more expressive for different downstream tasks [2] (As shown in Fig. 1 (c) in the attached PDF).\n\n\n**References.**\n\n[1] NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs.\n\n[2] Adaprop: Learning adaptive propagation for graph neural network based knowledge graph reasoning.\n\n[3] A* net: A scalable path-based reasoning approach for knowledge graphs.\n\n[4] Less is More: One-shot Subgraph Reasoning on Large-scale Knowledge Graphs.\n\n[5] Unifying large language models and knowledge graphs: A roadmap.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJUyOtwhQb", "reviewer": "Reviewer_89jh", "summary": "A clustering-guided anchor-based efficient KGE method is proposed in the paper. Concretely, it takes advantages of the clustering, which can be treated as an effective sampling way compared to random selected. Based on such idea, the authors apply the mechanism to different backbones and different downstream tasks. The experimental results show their great performances compared to previous anchor-based KGE method, i.e., NodePiece.", "questions": "1. Although the clustering preparation is beneficial for anchor selection, it will also bring additional computational overhead. How can we optimize such overhead in the future?\n\n2. Can you explain the advantages more about relational clustering-based anchor selection strategy in the proposed model?", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is first seen in knowledge graph representation learning from my aspect. It is novel, which is also simple and proven effective as shown in experiments.\n\n2. The experiments are sufficient, which shows the performances of the model from six aspects. Besides, three KGE models are adopted as the basic backbones for comparison. Moreover, two downstream tasks are evaluated. All these experiments sufficiently support the main claim of the proposed model.\n\n3. The paper takes attempt to address an important task in KGE, but with a simple yet effective strategy which can be easily applied to other GNN based models, especially in real-world scenarios.", "weaknesses": "1. Authors should double check the paper for writing mistakes, such as Superiority in line 191, etc. \n\n2. As LLM rapid developed, I am curious about whether we can integrate LLM-based models to enhance your strategy.\n\n3. Experimental settings should be described more, such as description of compared baselines, explanation of evaluation metrics, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions! We respond to your questions carefully one by one carefully, and we hope our responses can address your concerns.\n\n**RQ1. Writing Issues.** \n\nThanks for your suggestions. I will reorganize our paper into shorter paragraphs, especially for the second and third paragraphs of the introduction and related works. We will also double-check our paper to revise typo mistakes, reference redundancy and unify the reference formats.\n\n**RQ2. Corporation with our RecPiece and LLMs.** \n\nThanks, and we will respond from from two aspects.\n\n**(1) Enhancing RecPiece with LLMs.** As shown in Fig. 1 of our manuscript, there are four steps in our RecPiece. As the main goal for our RecPiece is to achieve a better trade-off between efficiency and effectiveness, integrating LLMs into our framework, especially for the embedding models, will raise the parameter redundancy except for the first step, i.e., feature preparation. Concretely, there are two potential ways to utilize LLMs. On the one hand, we can utilize LLMs for textual information encoding, as shown in Fig. 1 (a) in the attached PDF. Actually, we have already tried to leverage PLMs for it in Sec. 4.2.2 and Tab. 4. Compared to PLMs, LLMs are of better expressive capacity, which will lead to better representation. On the other hand, when applying our RecPiece to real-world scenarios, we can use LLM-based agents for real-time information retrieval and collection, as shown in Fig. 1 (b) in the attached PDF. It saves the expensive labor cost for information collection before feature preparation.\n\n**(2) Enhancing LLMs with RecPiece.** Our RecPiece can be treated as an efficient feature encoding method, which is an option for utilizing large-scale KGs in our real-world applications. With the generated embeddings for the structural knowledge, LLMs can be more expressive for different downstream tasks [1] (As shown in Fig. 1 (c) in the attached PDF).\n\n**RQ3. More Description on KGE Baselines, and Evaluation Metrics.**\n\nThanks for your suggestions. I will add the description of the baselines in our final version of the manuscript. As for the evaluation metrics, we have already introduced them in Appendix A.3.3. We will also add more descriptions of them in our final version. More details can be checked in RQ1 of Reviewer vJzB.\n\n**RQ4. Optimization on Redundant Overhead of Clustering-based Strategy**\n\nThanks for your valuable advice. The question you mention is a common one for all anchor-based KGE models, where the anchor selection procedure is inevitable yet crucial. In the future, we can integrate the anchor selection procedure with the model learning procedure. Besides, we only need to run the clustering-based anchor selection algorithm once. With better-quality anchors, better performances can be achieved. From this view, the redundant overhead for anchor selection is also acceptable.\n\n**RQ5. Advantages of our Relational Clustering-based Anchor Selection Strategy**\n\nThanks and we will answer your questions from three aspects, which are actually described in Sec. 3.6 of our manuscript.\n\n**(1)** Random and manual anchor selection in previous anchor-based models is highly dependent on the human experience. Compared to them, ours is more reasonable and learnable according to the representative ability of the cluster centroids.\n\n**(2)** Our RecPiece is developed based on the characteristics of KGs. Specifically, we perform clustering on features of triplets instead of entities since the knowledge units in KG are stored in triplets, which can also be easily characterized based on the relation type.\n\n**(3)** The hyper-parameter, i.e., cluster number, for clustering algorithms can be determined according to the attributes in KGs in RecPiece as the number of relation types. Thus, our anchor selection only contains one hyper-parameter, i.e., anchor number, which is inevitable and also needed by other anchor-based methods. Besides, other models even need resource-consuming grid-searching to get weights for different centrality measurement strategies on different KGs.\n\n**Reference**\n\n[1] Unifying large language models and knowledge graphs: A roadmap.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o0yBiNBjxZ", "reviewer": "Reviewer_vJzB", "summary": "The paper presents an interesting idea to make the existing KGE methods achieve better efficiency, termed RecPiece. It is proposed based on two characteristics, i.e., representative ability of cluster centroids and the descriptive ability of the relational facts. Complete experiments are carried out based on five datasets, two downstream tasks, and three KGE backbones, which shows its promising performances.", "questions": "Is it possible to extend your idea in inductive settings? \n\nI would recommend the authors discuss more about making use of multi-modal attributes for feature preparation in the future work.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is overall easy to follow, and readable.\n2.\tThe idea of the paper is interesting and practical. It makes use of the clustering strategy to generate the representative data first, and then, performs propagation based on it. Although such prototype ideas are appeared in other fields, like computer vision, etc. From my experiences, it is first applied for KGE. \n3.\tThe authors explain why they select relational facts for clustering, instead of directly using entity features for clustering. I really like the thoughts behind it that the authors carefully think about the differences between KGs and other general graphs, and take use of it.\n4.\tThe experiments are complete. As I mentioned in summary above, various experiments are carried out based on five datasets over two downstream tasks (4 for link prediction and 1 for entity classification). Besides, three KGE backbones, RotatE, CompGCN, and AutoSF, are leveraged. Moreover, experiments demonstrate the performances of their model from 6 aspects.", "weaknesses": "You should explain more about the compared baselines in your experiment settings, even with one sentence. It can make the reader better understand your paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions! We respond to your questions carefully one by one carefully, and we hope our responses can address your concerns.\n\n**RQ1. More Description on KGE Baselines, and Evaluation Metrics.**\n\nThanks for your suggestions. I will add the description of the baselines in our final version of the manuscript. As for the evaluation metrics, we have already introduced them in Appendix A.3.3. We will also add more descriptions of them in our final version. Concretely, the descriptions of the compared KGE baselines are presented as follows:\n\n**RotatE** defines each relation as a rotation from the source entity to the target entity in the complex vector space, enabling it to model and infer various relation patterns, including symmetry, antisymmetry, inversion, and composition relations. \n\n**COMPGCN** encodes entities and relations jointly by using various composition operators from KGE techniques, addressing the issue of over-parameterization in GCNs. \n\n**AutoSF** proposed an algorithm that can automatically design and discover optimal scoring functions of the KGE model. Through a progressive greedy search algorithm, AutoSF is able to design promising KGE scoring functions effectively from a vast search space. \n\n**TransE** is a translation-based KGE model that aims to model inversion and composition relations. Inspired by the translation invariance in the word2vec model, TransE tries to make h + r≈t, where h, r, and t represent the head entity, relation, and tail entity in a triplet, respectively.\n\n**DistMult** assumes that all relations in the KG are symmetric and represent them as block-diagonal matrices. Such relation representation mechanism, combined with simple dot product operations improves the efficiency of triplet evaluation. \n\n**ComplEx** uses complex vectors instead of real vectors to represent the embeddings of entities and relations, which allows the model to distinguish between symmetric and asymmetric relations.\n\n**PairRE** uses paired vectors to represent each relation, allowing the margin in the loss function to adjust adaptively. Thus, PairRE can express more complex relations, such as sub-relations. \n\n**TripleRE** combines projection and translation operations. Specifically, the representation vectors of the head and tail entities are first projected and then translated to obtain the relation representation. This method enriches the expression of relations, enabling the model to handle complex relations.\n\n**NodePiece** proposed a efficient and plug-and-play node selection mechanism for KGE models. Specifically, NodePiece is inspired by WordPiece from the field of natural language processing, which are able to represent large-scale knowledge graphs using only fewer entity embeddings, while also enhancing the generalization performance of the model at the same time.\n\n**LRE** is a high-efficiency method that utilizes tensor decomposition to enhance the parameter efficiency of KGE models. Specifically, rather than decomposing the observed 3D tensor directly, LRE decomposes the entity embedding matrix to low-rank matrices.\n\n\n**RQ2. Inductive Settings** \n\nThanks for your question. The current version of our RecPiece cannot handle the inductive settings, as the clustering-based anchor selection strategy is performed on the preprocessed features of the whole graphs. However, unseen entities or relations exist in inductive settings, where will lead the bias for anchor selection during training. But thanks for your valuable comments, we will try to work on it in the future.\n\n**RQ3. Utilization of Multi-modal Attributes** \n\nThanks for your valuable advice, and we will answer your questions as follows.\n\n**(1)**\tSure, it is definitely possible to utilize other multi-modal attributes, and we have already considered it. Specifically, the performance comparison and discussion are present between different prepared features from both structural information and textual information in Sec. 4.2.2 and Tab. 4 of our paper. \n\n**(2)**\tWe further prepare the clustering features from visual information. Specifically, we make use of the multi-modal version of the FB15k-237 datasets from [1]. Then, we further leverage ViT [2] to generate the visual features of each entity. According to the Tab.1 in the attached PDF, we can also see that our RecPiece can achieve better performance with the visual attributes, which further demonstrates the scalability and effectiveness of our method.\n\n\n\n\n**References.**\n\n[1] A survey of knowledge graph reasoning on graph types: Static, dynamic, and multi-modal\n\n[2] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the SAC, AC, and PCs for their efforts and constructive comments, which are helpful in further improving the quality of our manuscript. We respond to your questions carefully one by one carefully, and we hope our responses can address your concerns.

Note that there are two tables and one figure in the attached PDF, corresponding to RQ3 and RQ4 for Reviewer z3Fe, RQ1 and RQ4 for Reviewer GCgr, RQ2 for Reviewer 89jh, and RQ3 for Reviewer vJzB.

Meta-review: This submission proposes a modification to NodePiece [18] by changing the clustering scheme and encodings shown in eqns. (1) and (2) in [18] to incorporate a relation representation that is based on the (head, tail) pairs connected by the relatio.  This submission is not self-contained; I eventually had to go and read [18] (a more creative and much better-written paper) to make sense of the current submission.  The modification made to [18] feels relatively minor.  The (head, tail) based representation of relations has been used for a few years (see citation below).  There is some improvement in KG completion (0.3-5.2 relative percent) and entity classification (0.3%), which are not earth shattering.   


Beyond numeric scores, this submission sinks or swims based on its marginal contributions beyond [18] (NodePiece).  Table 1 has a succinct summary.  Observe that model size savings from RotatE to NodePiece are 10x for FB15k237, 8x for WN18RR, 20x for CoDEx-L, 30x for YAGO3-10.  In comparison, the model size savings from NodePiece to RecPiece are 9% for FB15k237, 0% for WN18RR, 17% for CoDEx-L and 4% for YAGO3-10.  Similar observations hold for Table 2.  These are orders of magnitude apart, so most of the heavy lifting was done by the key innovation in NodePiece.

Accuracy improvements reported in Tables 3 and 4 are also very modest. 

In summary, it is possible to argue that both the technique and benefit beyond NodePiece are modest, except for the OGB WikiKG dataset, which was likely the authors' primary target.

If the panel considers the incremental progress still important to publish and report, they should also consider the fact that the manuscript is rather poorly written.


Only that last part is fixable. Some suggestions follow.

L41 This is the first occurrence of the word "propagation" after the abstract.  The typical reader might benefit from a 1-2 sentence disclosure of what "propagation" means in this context.  What exactly are you propagating "from an intact set of all of the entities to an anchor set of some representative entities"?

To keep the manuscript self-complete and not require the reader to first read [18], "anchor based sampling" should be explicated further in the intro itself.  E.g. up to the end of page 2, we do not know if an anchor is an entity, or an aggregation over entities, or a relation type, or a representative relation type.  Unfortunately, without the assistance of concrete notation, the intro fails to give anything beyond a remote hint of the technique.

The caption (not distant text) of Figure 1 should include these pieces of info:
* Before entering $p(\cdot)$, what are node representations?  Do they depend on any textual aliases of nodes?
* Is the feature preparation capable of gradient-based updates, or is a fixed feature computation function?
* Are the colored vectors between the applications of $p(\cdot)$ and $g(\cdot)$ mutable through the training process or fixed?
* In the topmost row, $B$ get a shade of blue, but in the third row, it gets an orange shade.  What is the significance of this?
* If $|\mathcal{R}|$ is the number of distinct relation types, it seems there is one point cloud for each relation type, characterized by the features of the entity-pairs participating in that relation type.
* After application of $\phi _a$ we see cluster centroids marked.
* After application of $\phi _b$ we see some nodes in the original graph acquire a color, but some other nodes do not.  Explain why.
* How is $f(\cdot)$ a _propagation_?  Around this point the diagram gets too murky to follow.

L109 Link prediction in the context of KGs is just KG inference or completion, but what is node (entity) classification?  What are labels?

L117 "Thus, we selected the features of triplets as the clustering features in this paper." ― what _are_ the features of the triplets?

L122 and we still have no idea what "feature propagation" means.

Between equations (1) and (2) there are various sources of confusion.

L132 "first generate the embeddings of entities and relations" ― how?  Must we inspect your code to find out this critical detail?

Nit: If you will be using $h$ for embedding, do not use $h$ for head, use subject and object instead.

L133-L134 is equally problematic. What is a "normalized embedding"?  Assuming $\bar{h} _{\text{head}}$ and $\bar{h} _{\text{tail}}$ have the same width, and normalization does not change the length of a vector, how do you "sum" these with $\bar{h} _r$?

I know what you are doing here, just that the notation is a disaster.

On L134, why isn't $n$ also attached to $e _h$ and $e _t$?

On L135 you are about to define $\mathbf{H} _t$ but then suddenly switch to $\mathbf{H} _\mathcal{G}$ in the lhs in eqn. (2).  Is this a typo?

In eqn. (2), $\bigoplus$ means concatenation.  Depending on the popularity of the relation type involved, you therefore get representations of variable lengths?

Using multisets of (subject, object) pair embeddings to represent a relation has been known in the KG literature for some time:
https://ieeexplore.ieee.org/document/10191597  (see Eqn (4))

In eqn (3), is there any reason to set the number of clusters to precisely $|\mathcal{R}|$?

L151 Nit: Cosine is a similarity, not a distance.

L157 So at this point, for every relation type $i \in [ | \mathcal{R} | ]$, you collect $m _i$ triples, for a total of $\sum _i m _i$ exemplar triples?  Why possible different $m _i$s for different relation types (eq. 6)?

L166 What _are_ "the features of each entity $e _i$"?  The reader should not wait for section 4.2.2.

In Section 3.5 of your submission, you mention propagation in the context of [18] but the NodePiece paper has zero occurrence of the word "propagation".
Decision: Accept (poster)