Decentralized Knowledge Graph Representation Learning
Authors: Lingbing Guo
Weiqing Wang
Zequn Sun
Chenghao Liu
Wei Hu
Venue:  (2021)
Keywords: Representation Learning
Knowledge Graph
Entity Alignment
Knowledge Graph Completion
Knowledge Graph Embedding

Abstract: Knowledge graph (KG) representation learning methods have achieved competitive performance in many KG-oriented tasks, among which the best ones are usually based on graph neural networks (GNNs), a powerful family of networks that learns the representation of an entity by aggregating the features of its neighbors and itself. However, many KG representation learning scenarios only provide the structure information that describes the relationships among entities, causing that entities have no input features. In this case, existing aggregation mechanisms are incapable of inducing embeddings of unseen entities as these entities have no pre-defined features for aggregation. In this paper, we present a decentralized KG representation learning approach, decentRL, which encodes each entity from and only from the embeddings of its neighbors. For optimization, we design an algorithm to distill knowledge from the model itself such that the output embeddings can continuously gain knowledge from the corresponding original embeddings. Extensive experiments show that the proposed approach performed better than many cutting-edge models on the entity alignment task, and achieved competitive performance on the entity prediction task. Furthermore, under the inductive setting, it significantly outperformed all baselines on both tasks.

Under review as a conference paper at ICLR 2021
DECENTRALIZED KNOWLEDGE GRAPH REPRESENTATION LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Knowledge graph (KG) representation learning methods have achieved competitive performance many KG-oriented tasks, among which the best ones are usually based on graph neural networks (GNNs), powerful family of networks that learns the representation of an entity by aggregating the features of its neighbors and itself. However, many KG representation learning scenarios only provide the structure information that describes the relationships among entities, causing that entities have no input features. In this case, existing aggregation mechanisms are incapable of inducing embeddings of unseen entities as these entities have no pre-defined fea- tures for aggregation. In this paper, we present decentralized KG representation learning approach, decentRL, which encodes each entity from and only from the embeddings of its neighbors. For optimization, knowledge from the model itself such that the output embeddings can continuously gain knowledge from the corresponding original embeddings. Extensive experi- ments show that the proposed approach performed better than many cutting-edge models on the entity alignment task, and achieved competitive performance on the entity prediction task. Furthermore, under the inductive setting, it significantly outperformed all baselines on both tasks.
design an algorithm distill
INTRODUCTION
Knowledge graphs (KGs) support many data-driven applications (Ji et al. 2020). Recently, learning low-dimensional representations (a.k.a. embeddings) of entities and relations in KGs has been increasingly given attentions (Rossi et al., 2020). We find that existing models for KG representa- tion learning share similar characteristics to those for word representation learning. For example, TransE (Bordes et al. 2013), a well-known translational KG embedding model, interprets a triple (eI,T,e) as e + e2, where el, 62, denote subject, object and their relationship, respectively, and the boldfaces denote the corresponding embeddings. If we view éj as a word in sentences, and 62 well many other objects of é₁ the context words, then TransE and many KG embedding models (Wang et al.) 2014 Dettmers et al. 20118] 8, Nguyen et al. 2018 Kazemi & Poole 2018 Sun al. 2019), learn representations in form simaliar to that used in Skip-gram (Mikolov al. (013a), where the input representation is learned to predict the context (i.c., neighbors) representations.
Recently, many graph neural networks (GNNs) based models for KG representation learning (Wang ct al. 2018 Schlichtkrull et al. 2018 Cao et al. 2019 Wu et al./ 2019; Sun et al. [2020; Vashishth al.
2020) have achieved state-of-the-an performance in KG-related tasks such entity alignment and entity prediction. Those models learn KG representations in a CBOW (continuous bag-of-words, (Mikolov et al. 2013a) manner, which the context entities are aggregated predict the target. But they also consider the representation of an entity itself when aggregating the neighborhood informa- tion. This nature prevents those models (e.g., GCN (Kipf & Welling 2017) and GAT (Velickovic et al. (2018) to be generalized to represent unseen entities. In many cases, the entities in prevalent KG-related tasks do not have self features. This motivates us to learn entity representations from and only from their context neighbors.
We propose a decentralized KG representation learning approach. decentRL. The key idea of decentRL is to decentralize the semantic information of entities over only their neighbors (i.e., distributed context vector in CBOW (Mikolov et al.] (013b)), which can be easily implemented by representing each entity through averaging its neighbor embeddings. In this paper, we look for

---

Under review as a conference paper at ICLR 2021
more efficient but still simple way to realize this concept on the most popular graph attention network (GAT) (Velickovic et al. 2018). as well as its many variants (Sun et al., 2020: Vashishth et al. 2020). We illustrate the methodology by decentralized attention network (DAN). which is based on the vallina GAT. DAN is able to support KG representation learning for unseen entities with only structure information, which is essentially different from the way of using self features (e.g.. attribute information) in the existing graph embedding models (Hamilton et al. 2017 Bojchevski & Günnemann 2018 Hettige et al.) 2020). Furthermore, the neighbors in DAN serve as an integrity to give attentions, which means that DAN is more robust and more expressive compared with conventional graph attention mechanism (Velickovic et al. 2018).
Another key problem in decentralized KG representation learning is how to estimate and optimize the output embeddings. If we distribute the information of an entity over its neighbors, the original embedding of this entity e; also learns how to effectively participate in the aggregations of its different neighbors conversely. Suppose that we have obtained an output representation g: from DAN for entity e;, we can simply estimate and optimize g: by aligning it with e,, But directly minimizing the L1/L2 distance between g, and e; may be insufficient. Specifically, these two embeddings have completely different roles and functions in the model, and the shared information may not reside in the same dimensions. Therefore, maximizing the mutual information between them is better choice. Different from the existing works like MINE (Belghazi al., 2018) InfoNCE (van den Oord et al. 2018), in this paper, we design a self knowledge distillation algorithm, called auto-distiller. It alternately optimizes g, and its potential target e,, such that g, can automatically and continuously distill knowledge from the original representation e across different batches.
The main contributions of this paper are listed as follows. (1) We propose decentralized KG representation learning, and present DAN as the prototype of graph attention mechanism under the open- setting. (2) We design an efficient knowledge distillation algorithm to support DAN for generating representations of unseen entities. (3) We implement an end-to-end framework based on DAN and auto-distiller. The experiments show that it achieved superior performance on two prevalent KG representation learning tasks (i.e., entity alignment and entity prediction), and also significantly outperformed those cutting-edge models under the open-world setting.
BACKGROUND
Knowledge Graph. A KG can be viewed as multi-relational graph, in which nodes represent entities in the real world and edges have specific labels to represent different relationships between entities. Formally, we define KG as a 3-tuple g = (E, R. T)), with & and R denoting the sets of entities and relationships, respectively. T is the set of relational triples.
KG Representation Learning. Conventional models are mainly based on the idea of Skip-gram. According to the types of their score functions, these models can be divided into three categories: translational models (e.g., TransE (Bordes et al., (2013) and TransR (Lin et al.. (015a)), semantic matching models (c.g., DistMult YYng et al. 2015) and ComplEx (Trouillon et al. 20166) and neural models (e.g., ConvE (Dettmers et al. (018)) and RSN (Guo al. 2019)). We refer interested readers to the surveys (Wang et al. 2017 Ji et al. 2020) for details. Recently, GNN-based models receive great attentions this field, which are closely related to this paper. Specifically, R-GCN (Schlichtkruil et al. 2018), AVR-GCN (Ye et al. 2019) and CompGCN (Vashishth et al. 2020) introduce different relation-specific composition operations to combine neighbors and the corresponding relations before neighbor aggregation. RDGCN (Wu et al. 2019) refactors KGs as dual relation graphs (Monti et al. 2018) where edge labels are represented as nodes for graph convolution. All the aforementioned GNN- models choose GCNs and/or GATs to aggregate the neighbors of an entity, in which an identity matrix is added to the adjacency matrix. This operation is helpful when elements have self features, but poses problem in learning the representations of unseen entities where no self features are attached to them. Differently, decentRL fully relies on the neighbor context to attend to the neighbors of each entity in linear complexity, which is efficient and easy to be deployed.
Entity Alignment. Entity alignment aims to find the potentially aligned entity pairs in two different KGs G1 (E1, R1, T1) and G2 = (E2, R2, T2), given a limited number of aligned pairs as training data S C E x Eg. Oftentimes, G1, 92 are merged to a joint KG G = (E, R, T), which enables the models learn representations in unified space.
2

---

Under review as a conference paper at ICLR 2021
Entity Prediction. Entity prediction (a.k.a. KG completion (Bordes et al. 2013 seeks to find the or (E1.
missing subject é1 or object 62. given an incomplete relation triple (?, r, E2)
It is worth noting that the performance complex deep networks, as it relies on the predictive ability rather than the embedding quality (Guo) models cannot obtain promising results in entity align- ment (uuo et al. 2019 Sun et al. 2020). Differently, entity alignment directly compares the distance of learned entity embeddings, which clearly reflects the quality of output representations. Few models demonstrate consistently good performance on both tasks, whereas decentRL is capable of achieving competitive, even better, performance compared with respective matte-of-te--as models.
the entity prediction task may be greatly improved by
al. 2019). Hence, many cutting-edse
DECENTRALIZED REPRESENTATION LEARNING
In the decentralized setting, the representation of an entity e, is aggregated from and only from its neighbors N₂ {ei. 62 ...., (NN,). ,, As it may have many neighbors that unequally informa- tive (Velickovic et al. 2018), involving attention mechanism is a good choice.
3.1 GRAPH ATTENTION NETWORKS
We start by introducing the Graph attention network (GAT) (Velickovic et al. 2018 which leverages linear self attention to operate spatially close neighbors. For entity e GAT aggregates the representations of its neighbors N; and itself into a single representation c; as follows:
C; =
Σ
Oig Wej.
eJENUU
where asj the learnable attention score from e; to Bj and W the weight matrix. To obtain Oizs linear attention mechanism is used here:
Oi = softmax o (a [ W2e,))),
where a is weight vector to convert the concatenation of two embeddings into scalar attention score, and denotes the concatenation operation. W, and W₂ are two weight matrices. a the activation function, usually being LeakyRel.U (Xu et al. 2015). GAT computes the attention score of an entity e, to its neighbors in linear complexity, which is very efficient when being applied to large-scale graphs.
3.2 DECENTRALIZED ATTENTION NETWORKS
Intuitively, if is the embedding of an unseen entity, it is rarely useful in computing the attention scores (as it is just a randomly initialized vector). Thus, purely relying on its neighbors may good choice. Specifically, obtain the decentralized attention scores, one may simply sum all the attention scores from other neighbors afş = softmax ENNA Okz). However, it would lead to a problem that this sum only represents the attention of each neighbor to ej. In this case, a high attention score from one neighbor eą to ej can dominate the value of ofy. but does mean that e, is more important for e,. Therefore, all neighbors should act as an integrity in giving attentions.
Towards this end, we propose decentralized attention networks (DANs). Formally, to obtain the decentralized attention weight aij. we have to feed the attention layer with two types of input: the neighbor context vector n; (i.e., query), and the candidate neighbor embedding e, (i.e., key and value). Separately controlling the iterations of these two variables in multi-layer model is evidently inefficient. Instead, we realize this operation by involving second-order attention mechanism. For layer k. DAN calculates the decentralized attention score af as follows:
softmax(xaaa. (W/k ( 2
k
where d.- k- denote the output embeddings of layer k for and of layer k for ej. respectively. we regard d as the neighbor aggregation of layer - for e,, then d 2 is exactly the embedding of ej used in summing d In this case, at, can represent the attention weight of
3

---

Under review as a conference paper at ICLR 2021
GAT
DAN
Linsar Alletion Layer
larget entity layer k
larget exttty layer l
Attecttion Layer
Ateuzion Layer
Softnax
Derse3
value
Weight
key & value
query
key & value
quary
Conceletalion
self-containndd neighbors
target eatity
Dercel
Derse2
neighbors
larget, entily
query
key
gt-1
Figure 1: Comparing graph attention network (GAT) with decentralized attention network (DAN).
e's neighbor context to ej. Then, can obtain the output of layer k by:
d =
Σ CGEN;
* ij
(4)
is worth noting that we perform convolutions on layer k as the score at, is attended to the neighbor representations in layer k - 2. This keeps the consistency and ensures the output representations are consecutive. Also, it enhances the correlation output in different layers, and forms the second-order graph attention mechanism.
For the first layer of DAN, we initialize aœ and dj² as follows:
1 N
d
Σ W d ejEN:
(S)
aggregator to obtain the decentralized embedding d, of layer 0, but other aggregators like pooling may be employed as well. This simple mean aggregator can also be regarded as CBOW model with dynamic window size. For the architecture and implementation of DAN, please refer to Appendix A
Here, simply
3.3 INSIGHT OF DAN
We compare GAT with DAN in Figure Although the attention layers levearged by DAN and GAT are identical, the decentralized structure has two significant strengths:
Inductive representation learning. In GAT, the self embedding e; participates in the calculation of attention scores (Equation]) [ and the aggregation of neighbors (Equation 2 Therefore, when es an open entity, its embedding is completely random-initialized. In this case, the attention scores computed by GAT are almost meaningless. By contrast, DAN generates the embedding of without requirement of its embedding throughout. Such characteristic enables DAN to induce embeddings on unseen entities.
Robustness. When calculating the attention scores for an entity e,, GAT only takes e; query, the importance of other neighbors is overlooked, which may lead to biased attention computation. On the other hand. is generally known that most entities in KGs only has a small number of neighbors (Li et al., 2017). Due to the lack of training examples, the embeddings of these entities are not as informative as those with more neighbors. Therefore, they may be not capable of serving as queries for computing attention scores, causing that GAT cannot obtain reliable attention scores in some cases. By contrast, the queries in DAN are neighbor context vectors, which have richer semantics and also enable DAN compute the attention scores in unbiased way.
Furthermore, the computational complexity DAN is almost identical to that of GAT, except that DAN has an additional mean aggregator. From Figure I and Equation aggregator is evidently simpler than the linear attention layer, which means that its computational complexity (both time and space) be almost overlooked. Therefore, DAN is efficient model.
find that such an
4

---

Under review as a conference paper at ICLR 2021
4 DECENTRALIZED REPRESENTATION ESTIMATION
The final output representation of DAN for can be optimized by minimizing the L1/L2 distance between gi and e; to enable self-supervised learning. Such distance estimation pursues precise match at every dimension of these two embeddings, but ignores the implicit structure information across different dimensions.
4.1 MUTUAL INFORMATION MAXIMIZATION
As mentioned in Section the original embedding e; also serves as one of neighbor embeddings in aggregating the decentralized embeddings of its neighbors, which implies that itself also preserves the latent information used to support its neighbors. Inspired by MINE (Belghazi et al., 2018), InfoNCE (van den Oord et al. 2018). and DIM (Hjelm al. 2019), in this paper do not try to optimize gi by reconstructing the original representation e;, Instead, we implicitly align them in a way of maximizing the mutual information T(g;,0) e;).
Specifically, we define a learnable function f: IR & R + R to estimate the mutual information density (van den Oord et al.) 2018) between g; and the copy of (the reason why using the copied vector will be explained shortly):
e;) W,e, by),
(6)
where D and O are the dimensions of the output and input representations, respectively. W,, by are the weight matrix and bias, respectively, é; denotes the copy of e, We expect that f(g, é,) is significantly larger than f(g,, é,) for i. Following InfoNCE, the objective can be written as:
f(g e.) Σejex, fgg
7 7gg é;)
log X:
where X; fei...... ,Ix,1 )
contains |X,| sampled negative entities plus the target entity e,- Maximizing this objective results in maximizing a lower-bound of mutual information between g: and é; (van den Oord et al. 2018).
4.2 AUTO-DISTILLER
Note that in Equations and 7 we actually use the copy of the original representations, which leads to completely different optimization process compared with existing works. Specifically, some methods like InfoNCE or DIM jointly optimize the two input the density function, as both variables are the output of deep neural models requiring to be updated in hack-propagation. But in decentRL, e; is just a randomly initialized vector, and its gradient Equation (7) may conflict with the gradient where it is taken its neighbors. On the other hand, such optimization operation also prevents e; from learning the neighborhood information at the bottom layer, and compels this variable to match ga.
input neighbor in learning decentralized representations of
To address this problem, we view e, as a teacher, and gi as student to learn from the teacher TTan al. 2020). Our aim is to let this teacher continuously gain more knowledge to teach the student, which we call it auto-distiller. Therefore, our final objective is:
f(g, é,) Eegex, f(g:,e,)
fgg, 0,) argmax Σ E log EesEx f(gj, 0%
argmax E log Ridf X.
(8)
e,
E
and it has two important characteristies:
Lemma (automatic distillation). Optimizing the first term of Equation foor entity e, naturally contributes to optimizing the second term for the neighbors of e, which means conventional mini- batch training procedure can be applied.
Lemma 2 (lower-bound), The mutual information between gi and é, still auto-distiler.
in
Proof. See Appendix B
5

---

Under review as a conference paper at ICLR 2021
Table
Result comparison of entity alignment on DBPISK.
ZH-EN
JA-EN
FR-EN
Models
Hits@
Hits 10
MRR
Hits@1
Hits@
MRR
Hits@1 1
Hits@ 10
MRR
AlignE (Sun
2018
0.472
0.792
0.581
0.448
0.789
0.563
0.481
0.824
0.599
RSN Guo et m. 2019
0,508
0.745
0.591
0.507
0.737
0.590
0.516
0,768
0.605
0.418
0.667
0.442
AliNet (Sun et al. 2020
0.539
0.826
0.628
0.549
0,831
0.645
0.552
0,852
0.657
decentRL
0.589
0.819
0.672
0.596
0.819
0.678
0.602
0.842
0.689
5 EXPERIMENTS
We evaluated decentRL on two prevalent tasks, namely entity alignment and entity prediction, for KG representation learning. As few existing models show state-of-the-an performance on both tasks, picked the probe the effectiveness decentRL, we also conducted ablation study and additional experiments. Limited by the length, please see Appendix C for more analytic results.
methods in respective tasks and compared decentRL with them. To
5.1 DATASETS
Entity Alignment Datasets. We consider the JAPE dataset DBP15K (Sun et al. 2017). which is widely used by existing studies. It includes three entity alignment settings, each of which contains two KGs different languages. For example, ZH-EN indicates Chinese-English alignment on DBpedia.
Entity Prediction Datasets. We consider four datasets: FB15K WN18, FB 1BKK22377 and WNI8RR (Bordes et al., 2013) Dettmers et al.) 2018 Toutaaova & Chen 2015). The former two have been used as benchmarks for many years, while the latter two the corrected versions, as FB15K and WN18 contain a considerable amount of redundant data (Dettmers et al.) 2018).
5.2 EXPERIMENT SETUP
For both tasks, we initialized the original entity embeddings, relation embeddings and weight matrices with Xavier initializer (Glorot & Bengio 2010).
To learn ProssKKG embeddings for the entity alignment task, we incorporated contrastive loss (unn ct al. 2020 Wang et al. 2018) to cope with aligned entity pairs S, which can be written as follows:
La Σ (1.))S
Ig 8ill+ + Σ a{> - gg 1y11]
(KJ')S
where S S are the positive entity pair set and sampled negative entity pair set, respectively. ||- denotes the I.2 distance between two embeddings. and À are hyper-parameters. By jointly minimizing two types of losses, decentRL is able to learn cross-KG embeddings for entity alignment.
Similarly, for entity prediction, we also need to choose a decoder to enable decentRL to predict missing entities (Vashishth et al., (220).. We chose two simple models, TransE (Bordes et al. 2013) and DistMult (Yang et 2015) for the main experiments, which are sufficient to achieve comparable performance against the state-of- atttr.rrt.
5.3 ENTITY ALIGNMENT RESULTS
Table depicts the entity alignment results on the JAPE dataset. We observe that: (1) decentRL. significantly outperformed all the methods on Hits@1 and MRR, which empirically showed the advantage of decentRL in learning high-quality representations. (2) The scores of Hits 10 of decentRL were slightly below those of AliNet. We argue that decentRL is purely end-to-end model, which did not incorporate any additional data augmentation used in AliNet (Sun et al. 2020) that may improve the Hits @ 10 results. Moreover, decentRL is much easier be optimized, it does not need to coordinate the parametrss of each part in the pipeline. Also, there no conflict to combine decentRI. and the data augmentation algorithm for further improvement.

---

Under review as a conference paper at ICLR 2021
(a) ZH-EN
[b)JA-EN
(c) FR-EN
Open decentil
0.8
8 Loss decentRL
D AliNel
0.6
(177%
0
0.7
(
Loss AliNet
0.4
Open GAT
Loss GAT
0.2
He91
Her
MRR
He1
Her10
MRR
H@a1
Hea10
MRR
Figure 2: Open entity alignment results DBP15K. Bars with dotted lines denote the performance drop compared with the corresponding results in the non-open setting. The same to the following.
Table 2: Entity prediction results FB 15K37737
CompSCN+Ceas
0.254
Models
TransE
DistMult
CompCCN-DisMMult
0,252
6,066
Hitsor
MR
decemtL +DistMult
0,256
Raw
0.465
357
0.294
0.419
354
01.441
CompGCN-Trnnsl
0211
0.120
+DGCCN
decenRI 4TresE
0.255
* R-GCN
0.443
325
0.281
0.499
230
0.324
W-GCN
0.444
1,520
0.267
0.504
229
0.324
0.0
0.1
0.2
0.3
CompGCN
0.515
233
0.337
0.518
200
0.338
decentRl 0.521
1.99
0.334
0,541
151
0.350
Figure 3: MRR results on open FB15K-237.
Raw denotes the original results of the decoders.
We also evaluated the graph-based models in the open-world setting. Specifically, we first split the testing entity set into two subsets, namely known entity set and unknown entity set. Then, for those triples in the training set (in the non-open- non-open-world setting, all triples used in training) containing unknown entities, we moved them to the testing triple set, which were only available during the testing phase. We followed classical dataset splitting used in the entity alignment task, with 20% of entities in the original testing set are sampled as open entities. Table 6 in Appendix C.I compares the datasets before and after re-splitting.
The experimental results are shown in Figure 2 We can find that decentRL outperformed GAT and AliNet (the second-best model) on all metrics. Although its performance slightly dropped compared with that in the close setting, the results of others (especially GAT, which only uses self representation as "query") suffered more under this open-world setting. Overall, decentRL is capable of achieving faate-of-the-al performance on both open and conventional entity alignment tasks.
5.4 ENTITY PREDICTION RESULTS
We also evaluated decentRL on the entity prediction task, in comparison with different GNN-based models: D-GCN (Marcheggiani & Titov 2017). R-GCN (Schlichtkrull al. 2018 W-GCN (Shang et al. 2019), and CompCiCN (Vashishth et al., 2020). The results on FB 151552237 are shown in Table 2, from which we observe that: (1) decentRL significantly outperformed all the other models for many metrics, especially MR (mean rank). This demonstrates that decentRL can learn better representations for both popular entities (valued by MRR metric) and long-tail entities (valued by MR metric) (2) decentRL boosted DistMult to achieve almost state-of of of-the-art performance on FB15K-237, The simpler model TransE, also gained great improvement on all metries. The reason may be that DAN discovered better aggregation weights and our auto-distiller continuously refined the output representations.
The corresponding results on open entity prediction are shown in Figure We added -tate-of-theeart yet more complicated observe that: decentRL + DistMult outperformed all the other models under this open-world setting, which verified its effectiveness in inductive learning with only structure data. decentRL TransE achieved the second-best performance, followed by CompGCN + ConvE. Overall, decentRL provided the decoders with better representations and supported them to achieve competitive and even better performance the cutting-edge models.
model CompGCN + ConvE the comparison, from which we
7

---

Under review as a conference paper at ICLR 2021
Table Entity prediction results on FB15K and WN18.
FB1SK
WN18
Methods
Hits@
@its
Hits@ 10
MRR
MR
Hits @
Hits@3
Hits@ 10
MRR
MR
TransE
0.297
0.578
0.749
0,463
0.113
0.888
0.943
0.495
DistMult
0,546
0.733
0.824
0.654
97
0.728
0.914
0.936
0.822
902
ComplEx
0,599
0.759
0,840
0.692
0,599
0.75y
0.840
0.692
ConvE
0.558
0.723
0.831
0.657
51
0.935
0,946
0.956
0.943
374
RotatE
0,746
0.884
0.797
40
0.944
0.952
0.959
0.949
309
RSN
0.722
0.873
0.78
0.922
0.953
0,940
TuckER
0.741
0.833
0.892
0.795
0.949
0.955
0.958
0.953
decentRL + Trans
0.633
0.771
0.856
0.715
40
0.736
0.904
0.954
0.824
255
decentRT. + DistMult
0.664
0.793
0.872
0.740
32
0.944
0.951
0.958
0.949
2559
decentRL + ComplEx
0.745
0.847
0.901
0.804
33
0.945
0.952
0.958
0.949
251
Table 4: Entity prediction results on FB15K-237 and WNI8RR.
FB15K-237
WNISRR
Methods
Hits@
Hits@
Hits@ 10
MRR
MR
Hits@1 1
Hits@
Hits@ 10
MRR
MR
TransE
0.465
0.294
357
0.501
0.226
3,384
DistMult
0,155
0.263
0.419
0.241
254
0.39
0.49
5.110
ComplEx
0.158
0.275
0.428
0.247
339
0.41
0.46
0.51
0.44
5,261
ConvE
0.237
0.356
0,501
0.325
244
0.400
0.440
0.520
0.430
4,187
RotatE
0.241
0.375
0,533
0.338
177
0.428
0.492
0.571
0.476
3,340
RSN
0,202
0.453
0,280
TuckER
0.266
0.394
0.544
0.358
0.443
0.482
0.526
0.470
CompGCN ConvE
0,264
0.390
0.535
0.355
197
0.443
0.494
0.546
0.479
3,533
CompGCN Trans
0.242
0.367
0.510
0.332
214
CompGCN DistMult!
0.249
0.368
0,515
0.337
199
CompGCN ConvEl
0.262
0.385
0.532
0.352
215
decentRL TransE
0.241
0.362
0.521
0.334
159
0.290
0.420
0.505
0.369
4,710
decentRL. DistMult
0.257
0.385
0.541
0.350
151
0.433
0.481
0.542
0.470
4,613
decentRI. + ComplEx
0.261
0.388
0.544
0.354
172
0.422
0.466
0.533
0.458
3,744
"I* denotes methods executed by the source code with provided best parameter settings.
Table 5: Ablation study of entity alignment DBPI5K (average of 5 runs).
ZII-EN
JA-EN
FR-EN
Models
Hits@
Hits@ 10
MRR
Hits@
Hits@ 10
MRR
Hits@
Hits@
MRR
decentRL + auto-distiler
0.589
0.819
0.672
0.596
0.819
0.678
0.602
0.842
0.689
decentRL + infoNCE
0.579
0.816
0.665
0.591
0.816
0.673
0.593
0.834
0.682
decentRI.
0.571
0.802
0.655
0.589
0.807
0.669
0.591
0.831
0.679
a
0.579
0,812
0.663
0.589
0.812
0.671
0.593
0.836
0.681
centRT.
0.544
0.791
0.632
0.561
0.799
0.646
0.560
0.820
0.654
The detailed results on entity prediction shown in TablesBand respectively. For the conventional benchmarks FB 15K and WN18 that have been widely used for many years, our decentRL with only simple decoders achieved competitive even better performance compared with those state-of-the-ar art models. Furthermore, decentRL greatly improved the best results on MR, as it can more efficiently aggregate neighbor information to learn high-quality representations for those "challenging" entities.
On the other hand, we find that the performance of decentRI. on FB 7B155-237 and WN18RR is not as promising as that Table although it still achieved the best Hits 10 and MR on FB15K-237. We argue that this may be caused by the insufficient predictive ability of simple decoders (Guo et al.) 2019). However, we currently do not plan to adapt decentRL to some complex decoders like ConvE, such complicated architecture can largely increase the time and space complexity. For example, CompGCN with ConvE needs to be trained at least two days on a small dataset like 1515K-237
Overall. the performance of some simple linear KG representation learning models (i.e., TransE, DistMult, and ComplEx) received great benefits from decentRL, and even outperformed some cutting-edge models.
8

---

Under review as a conference paper at ICLR 2021
(a)
ZHEN
(c) FR-EN
08
decerstEL @AliNe
deceriiLL DAlNe
B decessRL DAlliNe
0518
0,590
0.50
0.571
0.515
0.55
0.580
0.388
0.53
0528
0:088
0.3
0.58
(U5R)
0.561
0L580
0.59
0.516
0.590
0.53
0.576
0,535
(L3835
0.590
0.000
0:302
0,410
Input
14
Concat
Input
Coocat
Input
14
Conca
Figure 4: Hits @1 results of each layer and the concatenation. The results of AliNet are from (Sun
al. 2020). It has no I.3 and 1.4 scores its best performance was achieved by two-layer model.
5.5 COMPARISON WITH ALTERNATIVE MODELS
To exemplify the effectiveness of each module in decentRL, we derived a series of alternative models from decentRL and report the experimental results on entity alignment Table "centRL" denotes the model that used DAN but self-loop was added to the adjacency matrix.
From the results, observe that: all the models the table achieved state-of-the- of-thh--rt performance on Hits@ metric, as DAN which leverages all neighbors as queries can better summarize the neighbor representations of entities.
On the other hand, we also find that decentRL + auto-distiller outperformed all the other alternatives. The centralized model centRL+ + auto distilist had a performance drop compared with the decen- tralized one. The main reason is that entities themselves in centRI. also participated in their own aggregations, which disturbed the roles of the original representations. Please see Appendix C.2 for the corresponding results under the open entity alignment setting.
5.6 COMPARISON THE OUTPUT EMBEDDINGS OF EACH LAYER
We also compared the performance of each layer in decentRI. and AliNet. As shown in Figure 4 decentRI. consistently outperformed AliNet on each layer except the input layer. mentioned before, decentRL does not take the original representation of an entity input, but this representation can still gain knowledge in participating in the aggregations of its neighbors and then teach the student good as
(i.e., the corresponding output representation). The performance of the input layer that in AliNet, because the latent information this layer may not be aligned in each dimension.
On the other hand, we also observe that concatenating the representations of each layer in decentRL also improved the performance, with a maximum increase of 0.025 (0.023 for AliNet). Furthermore, IecentRI. can gain benefits from increasing the layer number, while the performance of AliNet starts to drop when the layer number larger than 2 (Sun et al. 2020).
6 CONCLUSION
In this paper we proposed decentralized KG representation learning, which explores new and straightforward way to learn representations in open-world, with only structure information. The corresponding end-to-end framework achieved very competitive performance on both entity alignment and entity prediction tasks.
REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. abs/1607.004450. 2016.
Layer normalization.
CoRR.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. Mutual information neural estimation. In ICML, 2018.
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of attributed graphs: Unsupervised inductive learning via ranking. In ICLR, 2018.

---

Under review as a conference paper at ICLR 2021
Antoine Bordes, Nicolas Usunier, Alberto Sarcia-Duráne Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, 2013.
Yixin Cao, Zhiyuan Liu, Chengjiang Li, Zhiyuan Liu, Juanzi Li, and Tat-Seng Chua. Multi-channel graph neural network for entity alignment. In ACL, 2019.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2D knowledge graph embeddings. In AAAI, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Lingbing Guo, Zequn Sun, and Wei Hu, Learning to exploit long-term relational dependencies in knowledge graphs. In ICML, 2019.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. NIPS, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Bhagya Hettige, Weiqing Wang, Yuan-Fang Li, and Wray L. Buntine. Robust attribute and structure preserving graph embedding. In PAKDD, 2020.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. A survey on knowledge graphs: Representation, acquisition and applications. CoRR, abs/2002.00388, 2020.
Seyed Mchran Kazemi and David Poole. Simple embedding for link prediction in knowledge graphs. In NeurlIPS, Montréal, Canada, 2018
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Furong Li, Xin Luna Dong, Anno Langen, and Yang Li. Knowledge verification for long-tail verticals. PVLDB, 10, 2017.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI, 2015a.
Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In EMNLP, 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- tions in vector space. In ICLR Workshop. 2013a.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In NAACL-HLT, 2013b.
Federico Monti, Oleksandr Shchur, Aleksandar Bojchevski, Or Litany, Stephan Günnemann, and Michael M. Bronstein. Dual-primal graph convolutional networks. CoRR, 000.//006.00770. 2018.
Dai Quoe Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Q. Phung. novel embedding model for knowledge base completion based on convolutional neural network. In NAACL-LLT, 2018.
Andrea Rossi, Donatella Firmani, Antonio Matinata, Paolo Merialdo, and Denilson Barbosa. Knowl- edge graph embedding for link prediction: A comparative analysis. CoRR, abs/2002.00819, 2020.
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In ESWC, 2018.
10

---

Under review as a conference paper at ICLR 2021
Chao Shang, Yun Tang, Jing Huang. Jinbo Bi, Xiaodong He, and Bowen Zhou. End-to-end structure- aware convolutional networks for knowledge base completion. In AAAI, 2019.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15, 2014.
Zequn Sun, Wei Hu, and Chengkai Li. Cross-lingual entity alignment via joint suribute-preserrvii embedding. In ISWC, 2017.
Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. Bootstrapping entity alignment with knowledge graph embedding. In IJCAI, 2018.
Zequn Sun, Chengming Wang, Wei Hu, Muhao Chen, Jian Dai, Wei Zhang, and Yuzhong Qu. Knowledge graph alignment network with gated multi-hop neighborhood aggregation. In AAAI, 2020.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In CVSC, 2015.
Théo Trouillon, Johannes Welbl, Sehastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In ICML, 2016.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 0t//1/077003433 2018
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha P. Talukdar. Composition-based multi-relational graph convolutional networks. In ICLR, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions Knowledge and Data Engineering, 29, 2017.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In AAAI, 2014.
Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang. Cross-lingual knowledge graph alignment via graph convolutional networks. In EMNLP, 2018
Yuting WU, Xiao Liu, Yansong Feng, Zheng Wang, Rui Yan, and Dongyan Zhao. Relation-aware entity alignment for heterogeneous knowledge graphs. In IJCAI, pp. 5278 5284, 2019.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. CoRR, 2015.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In ICLR, 2015.
Rui Ye, Xin Li, Yujie Fang, Hongyu Zang, and Mingzhong Wang. A vectorized relational graph convolutional network for relatonelatioaal network alignment. In IJCAI, 2019.
11

---

Under review as a conference paper at ICLR 2021
A DECENTRALIZED ATTENTION NETWORKS
A.1 ARCHITECTURE
illustrate a four-layer decentralized attention network (DAN) in Figure 5 Following the existing works AliNet (Sun et al. 2020), CompGCN (Vashishth al. 2020 etc. Yee al. 2019 Wu et al. 2019p, we also combine the relation embeddings in the aggregation. The original entity embeddings (i.e., g ²),, relation embeddings and weight matrices are randomly initialized before training. At step 0, we initialize g" with the original entity embeddings by mean aggregator. At step 1, g,g- and g" are fed into DAN. Then, we combine the hidden representations with relation embeddings (steps 3). Finally, we obtain the output of the first layer g' (step 4). Repeating steps 1-4, we can sequentially obtain the output the last layers.
layer-1
DecentArta Blockl
Relation Combination
DecentAtt
Relation Combonation
layer-3
DesearAtto
Relation
DesemAAm
Relaton Combination
Block2
Black4
layer-2
layer-4
Figure 5: Overview of a four-layer decentralized attention network (DAN). Best viewed in color. Grey, blue, light blue and orange nodes denote original entity embeddings, decentralized entity embeddings, hidden entity embeddings and relation embeddings, respectively. Taking the first layer as example, as DAN requires the output embeddings of two previous layers as input, we randomly initialize the original embeddings as g- (identical to d=1), and use appropriate aggregator to generate the initial decentralized embeddings go (identical to do).
A.2 IMPLEMENTATION DETAILS
Infrastructure. Following GAT Veeickovicc al. 2018), we adopt dropout (Srivastava al. 2014) and layer normalization (Ba et al., 2016) for each module in DAN. To fairly compare decentRL with other models, we do not leverage the multi-head attention mechanism (Vaswani et al.) 2017] Velickovic et al. 2018) which has not been used in other GNN-based models, although it can be easily integrated.
Furthermore, we consider residual connections (He et al.) 2016) between different layers of DAN to avoid over-smoothing, which we not only consider the output of the previous layer "residual", but also involve the output of the mean aggregator (i.e., g!) This can be written as follows:
g
g.
g.
(10)
For simplicity, in the rest of the paper, still use g to denote the output layer k.
Adaption to different tasks. For different KG representation learning tasks, we also consider different adaptation strategies achieve better performance.
12

---

Under review as a conference paper at ICLR 2021
For the entity alignment task, we follow the existing GNN-based models (Sun al. 2020 Wang ct al. 2018) to concatenate the output representation of each layer as the final output. We formalize
as tols
gi
(11)
On the other hand, the entity prediction task prefers the prediction ability rather than the learned representations (Guo et al. 2019). We only use the output of the last layer as the final output representation, which allows to choose larger batch size hidden size to obtain better performance. We write it as follows:
(12)
To enhance the predictive ability of decoders, here we only regard the mutual information- -baeed loss as a kind of regularization (which is similar to MINE (Belghazi et al. 2018),, and thus we re-seale the loss weight to 0.001.
B AUTOMATIC KNOWLEDGE DISTILLATION
B.1 INSIGHT
The existing works usually choose to jointly optimize two input variables the density function f, in which these two variables can be regarded as two different outputs of two models. For example, InfoNCE uses an encoder to obtain the latent representations and another model to summarize those representations to one context vector. This is similar to DeepInfoMax and DeepGraphInfo, which also leverage two models to obtain local features and summarize global features, respectively.
However, in case, the mutual information that we want to maximize is between the input and output of the same model, where the input vectors are just randomly initialized raw embeddings. We argue that jointly optimizing the original input e, with the output g in Equation (7) may drive e, to completely match g:-
To resolve this problem, we only the copy of e, when estimating the mutual information density between c; and gi. In other words, we do not update the gradient c; in Equation (7). leading to natural knowledge distillation architecture.
Specifically, we separately optimize e; and g; in different training examples or batches. The first step corresponding to the former part of Equation
fgg é.)
argmax log g.f X.
( Eojex, fgg eg)
(13)
Here, e, is served as "pre-trained" teacher model teach "student". Hence, the learnable parameters are and
As aforementioned, e; needs participate in learning the representations of its neighbors, during which it can gain knowledge to teach its student g:. This step is achieved by the latter part in Equation (8):
eg, log Exsxx, f(g )x, X,
argmax G
Σ
E
(14)
CJEN.
where our aim to find the optimal e, maximize the mutual information between the original and output representations of its neighbors.
B.2 THE NOWER-BOUND MUTUAL INFORMATION
We do not really need explicitly separate the training procedure into the two steps deseribed in Appendix B.I which is widely used in adversarial learning. Instead, this knowledge distillation mechanism can be automatically achieved during different mini-batches.
13

---

Under review as a conference paper at ICLR 2021
Specifically, if expand Equation
13
little bit, then we obtain:
argmax log f(G(N.), N. X;
(N,; 0,
(15)
where N; = feyle, N; is the original neighbor embedding set for e; and O denotes the parameters of our decentralized model G. As the optimal ( for the model depends on the neighbor representation set N and the optimal density function f also relies on the output of the model, is impossible to search all spaces to find the best parameters. In practice,
choose to optimize weaker lower-
on the mutual information / (g. ē;) (Tian et al. 2020). In this case, a relatively optimal neighbor
embedding c in Equation 15 is:
J(G(N,)) 0:)
argmax log X1
(16)
ΣeSex, f(G((N)),,)
and we have:
[[G(fe),.... e.....,.,....,.......... , (
E log T(g. e,le"))
(17)
Σ
X;
Eajex, (MG ) =[(g;, é.)
(N;))N)
S E log
(18)
X.
$ T(g. è.)) + log([X;])
(19)
[gg,,))) é;):
(20)
where denotes the optimal setting for the corresponding parameters. Equations (18 and 19 are the conclusion of InfoNCE, given that |X,] is large enough. The above equations suggest that optimizing Cr can also lower-bound the mutual information without the requirement of other parameters being perfectly assigned.
Consider that the entity Ex may have more than one neighbor, we can optimize those cases together:
J(G(N)), ēÿ) ExEx F(G(N,),
argmax e=
Σ
E log
(21)
E
Evidently, the above equation is identical to Equation (4), which means that optimizing Equation (15 can subsequently contribute optimizing the original neighbor representations.
Therefore, the proposed architecture can automatically distill knowledge, in different mini-batches, from the original representations into the output representations.
C FURTHER ANALYSIS
C.I DATASET DETAILS
The detailed statistics of the entity alignment datasets are shown in Table Although we only set 20% of entities in testing removed from the training set.
open entities, there are actually than 20% of triples that were
For the details of datasets used in entity prediction, we suggest readers refer to (Bordes al. 2013 and (Dettmers et al. 2018).
C.2 ABLATION STUDY ON OPEN ENTITY ALIGNMENT
We also conducted an ablation study on the open entity alignment task, as shown in Table 7 The experimental results, in principle, are consistent to those conventional entity alignment. The proposed architecture (decentRL+ auto-distiller) still outperformed other alternatives. By contrast, the performance of the centralized model with auto-distiller dropped significantly, in comparison with that it almost has identical performance with decentRL+ infoNCE in Table 5 Another worth-noting point is that the gap Hits @ 10 narrowed in the open entity alignment task, which may be because the training data were shrunk considerably due to removing the corresponding triples referred to unseen entities.
14

---

Under review as a conference paper at ICLR 2021
Table Statistics of entity alignment datasets.
Original
Open
#Entities
#Relations
#Triples
#Train entity pairs
#Test entity pairs
#Truin triples
#Test triples
19,388
1,701
70,414
4,500
10,500
53,428
16,986
ZH-EN
19,572
1,323
95,142
4,500
10,500
72,261
22,881
19,814
1,299
77.214
4,500
10,500
57,585
19,629
JA-EN
19,780
1,153
93,484
4,500
10,500
69,479
24,005
19,661
903
105.998
4,500
10,500
79.266 87,030
26732
PR-EN
19,993
1,208
115,722
4,500
10,500
28692
Table 7: Ablation study open entity alignment. Average of 5 runs.
ZH-EN
JA-EN
FR-EN
Methods
Hits@
Hits@ 10
MRR
Hits@
Hits@ 10
MRR
Hits@1
Hits@
MRR
decentRL + auto-distileer
0.565
0.775
0.643
0.583
0.786
0.659
0.590
0.814
0.673
decentRL + infoNCE
0.557
0.775
0.637
0,574
0.785
0.652
0.583
0.811
0.666
decentRI.+122 1.2
0.552
0.770
0.632
0.574
0.782
0.650
0.581
0.806
0.664
ccntL
0.551
0.765
0.629
0.573
0.776
0.648
0.578
0.806
0.662
centRL
0.529
0.764
0.614
0.554
0.775
0.634
0.560
0.799
0.647
Table Performance decentRI. with different dimensions. Average runs.
ZH-EN
JA-EN
FR-EN
Hidden size
Hits@
Hits @ 10
MRR
Hits
Hits @ 10
MRR
Hits
Hits 10
MRR
64
0.429
0.644
0.508
0.474
0.673
0.547
0.468
0.704
0,554
128
0,511
0.726
0.590
0.541
0.745
0.617
0.535
0.773
0.623
256
0.560
0.785
0.643
0.578
0.791
0.657
0.578
0.817
0.665
512
0.589
0.819
0.672
0.596
0.819
0.678
0.602
0.842
0.689
C.3 IMPACT OF DIMENSIONS
We also evaluated decentRL under different settings of dimensions. The results are shown in Table With the increase of the input dimensions (i.e., embedding size). the performance of decentRL improved quickly, with dimension = 128 achieving comparable performance with the state-of- the-art methods (e.g., AliNet with 300 dimension) and outperforming them at dimension 256. Furthermore, decentRL can continually gain benefit from larger hidden sizes. Even when the dimension to 512, the improvement still significant.
15


Reviews:
[{"review_id": "ctPaPGaqhIL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents \"a decentralized KG representation learning approach\", named decentRL, which encodes each entity from and only from the embeddings of its neighbors. This approach can therefore account for new entities that have no known features to initialize its embeddings, but do have known links to other entities in the graph.\n\nThe main contributions:\n- The paper outlines decentralized attention network (DAN), an adaptation of graph attention network (GAT). GAT considers the direct neighbors of an entity in generating its embedding, and computes attention scores based on similarity between each neighbor and the focus entity. Assuming that no meaningful embedding may be available for new entities, DAT represent each entity via the embeddings of its direct 1-hop neighbors. It then considers its 2-hop entity neighbors for computing attention-based embeddings.\n- The paper further adapts the optimization process - alternately optimizing the representation of the target entity and its neighbors. \n\nExperimental results are presented on the task of entity alignment across Knowledge Graphs (KGs) and entity prediction (KG completion), using two datasets per task. Competitive performance is obtained in the general case, with improvements when new entities are considered.\n\nPros:\n- The approach is sensible, intuitive, and yields good results in practice.\n\nCons:\n- My main concern is the review of related work, which seems partial; e.g., the discussion is focused on GAT, but the best competing method in the results section is AliNet, which is nowhere described. Also, there exist other methods like DeepWalk which also rely solely on structure information.\n- The paper does not read easily\n- It is not clear if the code and data will to be released. At least, the dataset with train-test split should be released for comparison purposes.\n\nComments and typos:\n- can you motivate the open-world scenario, where only structure information is available for new entities, and it has no known features?\n- Table1: write Hits@1 instead of H@1\n- robuster --> more robust\n- Oppositely --> In contrast?", "extra_scores": {}, "extra_text": {}}, {"review_id": "TlDz5gMcX17", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for knowledge graph embedding based on graph attention networks (GAT). The key idea is to avoid using the information for a node (i.e., its representation vectors) when computing the attention weights for the neighbors of the node. The paper argues that this approach can better generalize to unseen nodes where no pre-defined features/information is available. As such, the paper does not include the representations for a node $e$ from prior layers in the aggregations to compute $e$'s representations in the next layers, leveraging the representation vectors of the nodes from prior layers to obtain attention weights for the current layer. The paper also proposes to a self-learning method to learn the parameters by optimizing the mutual information of the final and initial embedding vectors for the nodes. A distillation approach is also employed to use the initial embedding vectors as the teachers and the final embedding vectors as the students. The proposed method is applied to two downstream tasks, i.e., entity alignment and entity prediction, leading to competitive performance with many prior works (the learned node embeddings still need to be aligned using task-specific losses). Some experiments on unseen entities and ablation studies are also conducted to demonstrate the benefits of the proposed method. \n\nOverall, this paper is well written. It introduces an extension of GAT to address the unseen entity issue and the experiments seem to demonstrate its benefit. However, the motivation and approach of the paper should be better justified to make it more convincing. I have several comments/questions as follow:\n\n1. The technical novelty of the paper seems incremental as the DAN mechanism is a simple extension of GAT while mutual information maximization and the distillation are already applied in prior work.\n\n2. The paper seems to assume that for unseen entities, although pre-trained embeddings are not available, unseen entities are still connected to some seen entities so unseen entities' embeddings can still be obtained via the averages of the embeddings of the seen neighbors. As such, how do we handle two unseen entities that are neighbors of each other? More importantly, as the current method does not use any information specific to the nodes, e.g., node content (so only the connections of nodes are employed), can we just include the unseen entities in the graph and retrain the whole model? This is certainly more expensive, but as the paper is mainly considering downstream task performance, this might be a method to address unseen entities in this work. In general, as node embeddings are initialized randomly without considering node content in this work, the proposed method for unseen entities does not seem significant and convincing to me.\n\n3. Relatedly, as the node embeddings \\textbf{e}_i are randomly initialized, what kind of knowledge can \\textbf{g}_i expect to learn from considering \\textbf{e}_i as a teacher? In general, without the guidance from some specific downstream tasks, it is unclear which information the model would learns when the training finishes. Maybe the auto-distiller should be jointly trained with downstream tasks and the model can be better justified with the idea from information bottleneck? A discussion about the connection of the proposed method and information bottleneck is also helpful.\n\n4. How does the performance changes if we directly use \\textbf{e}_i (not its copy) in Equations (7) and (8)? This might help to better justify the model design.", "extra_scores": {}, "extra_text": {}}, {"review_id": "I34f-1lavra", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a GNN-based model for learning KG embeddings purely from the embeddings of its neighbours, which would enable learning entity embeddings for previously unseen entities. The model is based on a modified version of a graph attention network (GAT) which only considers the embeddings of neighbouring nodes.\n\nDespite the advantage of decentRL over existing approachs of its applicability on previously unseen entities and the results on entity alignment and entity/link prediction showing a lot of promise, I've found the paper quite hard to follow due to factual inaccuracies and English style and grammar issues, which is why I believe it is not ready for publication in its current form. I encourage the authors to revise the paper and resubmit to the next big ML/graphs conference.\n\n\nDetailed comments and questions below:\n\nSec. 1\\\nThe statement \"then TransE and many KG embedding models (Wang et al., 2014; Dettmers et al., 2018; Nguyen et al., 2018; Kazemi & Poole, 2018; Sun et al., 2019), learn representations in a Skip-gram (Mikolov et al., 2013a) manner\" is factually incorrect, since the data and learning objective for learning word embeddings differ greatly from those for learning KG embeddings. Analogical properties of word embeddings are an implicit bi-product of the skip-gram training objective, whereas in e.g. TransE this property is explicitly imposed on relation representations through its score function. The score function of other KG embedding models mentioned (e.g. ConvE, Dettmers et al. 2018 or SimplE, Kazemi & Poole 2018) is of a different type and does not even impose the analogical property of relations.\n\nSec. 3.1\\\n\"Intuitively, if e_i is the embedding of an unseen entity, it is rarely useful in computing the attention scores (as it is just a randomly initialized vector). Thus, purely relying on its neighbors may be a good choice.\" - Aren't the neigbours randomly initialised as well?\n\nSec. 3.2\\\nAfter being introduced, n_i is not used anywhere. What is its role in Eq. 3?\n\nSec. 5.4\n- As one of two main results sections, entity prediction results which include standard link prediction models (Tables 7 and 8) should be moved from the Appendix into the main body.\n- Given that TransE and DistMult haven't been state-of-the-art for quite a while, it would be interesting to see how decentRL performs with recent state-of-the-art link prediction models, such as TuckER or RotatE.\n\nOther comments:\n\nWriting quality should be improved, as the paper is hard to follow. Please have your submission proof-read for English style and grammar issues.\n\n===============================================================================================================\\\nAfter rebuttal:\\\nI have read the authors' response, but since the actual body of the paper has not changed much from the original submission, I stand by my original rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7-PpF95yDW3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary ===\n\nThis paper proposes a \"decentralized\" method for representation learning in knowledge graphs that doesn't explicitly depend on a learned embedding for the entity node of interest, e_i. Rather, the embedding for e_i is constructed in a distributed fashion (similar in motivation to the distributional hypothesis/skip-gram word embeddings) from its neighbors via a second-order attention mechanism. The main idea is that this is better for \"cold start\" problems in which unknown entities might have no features, which makes building any representation that explicitly depends on entity-centric features hard.\n\n=== Justification for Score ===\n\nWhile the method might indeed hold some promise, I find it a bit difficult to get excited about it. Furthermore, in the current version it's presentation and motivation is unclear and hard to follow. It is also not clear from the paper why this method should work better than simpler baselines (see concerns). \n\n=== Strengths ===\n\n- Relevant and timely topic, as leveraging knowledge graphs with initially unknown entities etc is an important problem (especially with constantly growing KGs).\n\n- Some mixed empirical results w.r.t. other compared models, but generally positive.\n\n=== Concerns ===\n\n- It's unclear to me why a purely \"decentralized\" representation is desired, especially if the experiments aren't purely on unknown entity settings. A natural baseline to me would seem to be a standard GCN/GAT with \"entity dropout\", i.e., during training time you introduce entities for which some redundancy must be gained by its neighborhood. Another approach would be to use a framework reminiscent of label propagation to deal only with imputing missing features at test time. Fully dropping all entity-specific features seems like overkill, and potentially harmful. After all, at test time, a majority of the KG will be known. \n\n- In general, in addition to the comment above, though I am not intimately familiar with their details, it appears that none of the considered baseline methods (AliNet, RSN, etc) are specifically designed to accommodate missing entities. As this is a major claimed contribution, it would be useful to either clarify this, or explain how it is handled differently in these networks. \n\n- I'm a bit perplexed by the consistently underperforming H@10 results relative to AliNet---and I'm not sure I find the justification in the paper (data augmentation) that convincing (can you clarify why this would explain worse performance @10, but not @1 or MRR?).\n\n- Overall, though there might be something here (I am willing to be convinced otherwise...), the paper fails to convince me of its significance at this stage. I feel that it would benefit greatly from an overall more compelling re-write.\n\n=== Minor Comments ===\n\n- In terms of readability, Table 2 is a bit too small.\n\n- Lemmas 1 and 2 don't add much to the paper in my opinion---I would recommend moving them fully to the appendix. \n\n=== Response After Rebuttal ===\n\nI thank the authors for their responses to my comments. After reading the response as well as the other reviews, I still stand by my original rating. I still find the motivation and empirical results non-compelling, given the current version of the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper brings interesting ideas (decentralized setting, auto-distillation) but it does not meet the very high requirements that a publication at ICLR requires.

Three main reasons for that:

1/ Motivation & justification: Ultimately the paper is advocating for a pure decentralized approach "which encodes each entity from and only from the embeddings of its neighbors" with the main motivation being to represent better on unseen entities at training. This is quite radical and leads to a complex model and training procedure for a benefit and justification that are not very clear. Are there that many unseen entities in general? What would periodically retrain the whole model do? The computational cost associated to DecentRL should be discussed with regards to that. Some implementation details in appendix A.2 seems rather critical and are not motivated.

2/ Missing comparisons and references: as noted by several reviewers, it would be helpful to have comparisons of other methods that are dealing with missing entities. Some much simpler heuristics could be tried for instance (retraining the model, averaging neighbors, etc.). A discussion with DeepWalk, that is really an adaptation of CBOW for KG should also be added.

3/ Clarity could be improved. Thanks to reviewers' comments, the clarity has increased but could still be worked on as noted by several reviewers. For instance, the analogy  with CBOW right in the intro is confusing: in the 2nd paragraph, CBOW is used as a common manner for methods that are limited, but in the 3rd paragraph, CBOW is also used as an intuition for DecentRL. Some content from supplementary material like the description in A.1 would add a lot of clarity if added earlier.

We encourage the authors to use the many comments from the reviewers to improve further the paper. 
Decision: Reject