PanRep: Universal node embeddings for heterogeneous graphs
Authors: Vassilis N. Ioannidis
Da Zheng
George Karypis
Venue:  (2021)
Keywords: Graph neural networks
universal node embeddings
node classification
link prediction
unsupervised learning

Abstract: Learning unsupervised node embeddings facilitates several downstream tasks such as node classification and link prediction. A node embedding is universal if it is designed to be used by and benefit various downstream tasks. This work introduces PanRep, a graph neural network (GNN) model, for unsupervised learning of universal node representations for heterogenous graphs. PanRep consists of a GNN encoder that obtains node embeddings and four decoders, each capturing different topological and node feature properties. Abiding to these properties the novel unsupervised framework learns universal embeddings applicable to different downstream tasks. PanRep can be furthered fine-tuned to account for possible limited labels. In this operational setting PanRep is considered as a pretrained model for extracting node embeddings of heterogenous graph data. PanRep outperforms all unsupervised and certain supervised methods in node classification and link prediction, especially when the labeled data for the supervised methods is small. PanRep-FT (with fine-tuning) outperforms all other supervised approaches, which corroborates the merits of pretraining models. Finally, we apply PanRep-FT for discovering novel drugs for Covid-19. We showcase the advantage of universal embeddings in drug repurposing and identify several drugs used in clinical trials as possible drug candidates.

Under review as a conference paper at ICLR 2021
PANREP: UNIVERSAL NODE EMBEDDINGS FOR HET- EROGENEOUS GRAPHS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning unsupervised node embeddings facilitates several downstream tasks such node classification and link prediction. A node embedding is universal if it is designed to be used by and benefit various downstream tasks. This work intro- duces PanRep, a graph neural network (GNN) model, for unsupervised learning universal node representations for heterogenous graphs. PanRep consists of a GNN encoder that obtains node embeddings and four decoders, each capturing different topological and node feature properties. Abiding to these properties the novel unsupervised framework learns universal embeddings applicable to different downstream tasks. PanRep can be furthered fine-tuned to account for possible lim- ited labels. In this operational setting PanRep considered for extracting node embeddings of heterogenous graph data. PanRep outperforms all unsupervised and certain supervised methods in node classification and link prediction, especially when the labeled data for the supervised methods is small. PanRep-FT (with fine-tuning) outperforms all other supervised approaches, which corroborates the merits of pretraining models. Finally, we apply PanRep-FT for discovering novel drugs for Covid-19. We showcase the advantage of universal embeddings in drug repurposing and identify several drugs used in clinical trials as possible drug candidates.
pretrained model
INTRODUCTION
Learning node representations from heterogeneous graph data powers the success of many down- stream machine learning tasks such as node classification (Kipf & Welling 2017). and link pre- diction (Wang et al.) 2017). Graph neural networks (GNNs) learn node embeddings by applying a sequence of nonlinear operations parametrized by the graph adjacency matrix and achieve state- of-the-art performance in the aforementioned downstream tasks. The era of big data provides an opportunity for machine learning methods to harness large datasets (Wu et al. 2013). Nevertheless, typically the labels in these datasets scarce due to either lack of information increased labeling costs (Bengio et al. 2012). The lack of labeled data points hinders the performance of supervised algorithms, which may not generalize well unseen data and motivates unsupervised learning.
Unsupervised node embeddings may be used for downstream learning tasks, while the specific tasks are typically not known priori. For example, node representations of the Amazon book graph can be employed for recommending new books as well as classifying a book's 's genre. This work aspires to provide wniversal node embeddings, which will be applied in multiple downstream tasks and achieve comparable performance to their supervised counterparts.
Although unsupervised learning has numerous applications, limited labels of the downstream task may be available. Refining the unsupervised universal representations with these labels could further increase the representation power of the embeddings. This can be achieved by fine-tuning the unsupervised model. Natural language processing methods have achieved sate-offthee-s performance by applying such fine-tuning framework (Devlin et al.) 2018). Fine-tuning pretrained models is beneficial compared to end-to-end supervised learning since the former typically generalizes better especially when labeled data limited and decreases the inference time since typically just few fine-tuning iterations typically suffice for the model to converge (Erhan et al. 2010).
This work introduces framework for unsupervised lcarning of universal node representations heterogenous graphs termed PanReg"] It consists of a GNN encoder that maps the heterogenous graph data to node embeddings and four decoders, each capturing different topological and node feature
Pan: Pangkosmios (Greck for universal) and Rep: Representation

---

Under review as a conference paper at ICLR 2021
PanRee-FF
PanRep
PanRep
Ma--
dooccer
Link predh chor Node cloced Iotor
ONN eroddyd
MRW
docoder
decicerr
Figure 1: Illustration of the PanRep (left) and PanRep-FT (right) models. The GNN encoder processes the node features X to obtain the embeddings H. The decoders facilitate unsupervised learning of H.
properties. The cluster and recover (CR) decoder exploits a clustering prior of the node attributes. The motif (Mot) decoder captures structural node properties that are encoded in the network motifs. The meta-path random walk (MRW) decoder promotes embedding similarity among nodes participating in a MRW and hence captures intermediate neighborhood structure. Finally, the heterogeneous information maximization (HIM) decoder aims maximizing the mutual information among node local and the global representations per node type. These decoders model general properties of the
graph data related node homophily (Gleich 2015 Kloster & Gleich 2014) node structural similarity (Rossi & Ahmed 2014 Donnat et al. 2018). PanRep is solely supervised by the decoders and has knowledge of the labels of the downstream task. The universal embeddings learned by PanRep are employed as features by models such as SVM (Suykens & Vandewalle, 1999) or
DistMult YYng et al. 2014) to be trained for the downstream tasks. To further accommodate the case where limited labels are available for some downstream tasks we propose fine-tuning PanRep (PanRep-FT), In this operational setting, PanRep-FT is optimized adhering to a susk-specific loss. PanRep can be considered as a pretrained model for extracting node embeddings of heterogenous graph data. Figure |
iillustrates the two novel models. The contribution of this work is threefold.
CL. We introduce a novel problem formulation of universal unsupervised learning and design a tailored learning framework termed PanRep. We identify the following general properties of the heterogenous graph data: (i) the clustering of local node features, (ii) structural similarity among nodes, (iii) the local and intermediate neighborhood structure, (iv) and the mutual information among same-type nodes. We develop four novel decoders to model the aforementioned properties.
C2. We adjust the unsupervised universal learning framework to account for possible limited labels of the downstream task. PanRep-FT refines the universal embeddings and increases the model generalization capability.
C3.We We compare the proposed models to state-of-the art supervised and unsupervised methods for node classification and link prediction. PanRep outperforms all unsupervised and certain supervised methods in node classification, especially when the labeled data for the supervised methods is small, PanRep-FT outperforms even supervised approaches in node classification and link prediction, which corroborates the merits of pretraining models. Finally, we apply our method on the drug-repurposing knowledge graph (DRKG) for discovering drugs for Covid-19 and identify several drugs used in clinical trials as possible drug candidates.
RELATED WORK
Unsupervised learning. Representation learning amounts to mapping nodes in an embedding space where the graph topological information and structure is preserved (Hamilton et al. 2017). Typically, representation learning methods follow the encoder-decoder framework advocated by PanRep. Never- theless, the decoder is typically attuned to a single task based on e.g., matrix factorization (Tang et al.
2015; Ahmed Ahmed al. [2013] Cao al. 20155 Ou et al. (2066), random walks (Grover Leskovee 2016 Perozzi et al. 2014). or kernels on graphs (Smola & Kondor 2003). Recently, methods relying on GNNs are increasingly popular for representation learning tasks (Wu et al. 2020). GNNs typically
rely on random walk- based objectives (Grover & Leskovec 2016 Hamilton et al., 2017) or on
maximizing the mutual information among node representations (Velicković et al. 20)8b). Relational
GNNs methods extend representation learning to heterogeneous graphs (Dong et al. 2017 Shi et al.)
2018 Shang et al. 2016).. Relative to these contemporary works PanRep introduces multiple decoders to learn universal embeddings for heterogeneous graph data capturing the clustering of local node features, structural similarity among nodes, the local and intermediate neighborhood structure, and the mutual information among same-type nodes.
2

---

Under review as a conference paper at ICLR 2021
Supervised learning, Node classification is typically formulated semi- supervise learning (SSL)
task over graphs, where the labels for a subset of nodes are available for training (Belkin et al. 2004).
GNNs achieve state-of-the-arr performance in SSL by utilizing regular graph convolution Kipf &
Welling 2017) or graph attention (Veličković et al.
2018a), while these models have been extended
in heterogeneous graphs (Schlichtkrull et al. 2018 8 Fu al. [020) Wang et al.] 2019h). Similarly, another prominent supervised downstream learning task is link prediction with numerous applications
in recommendation systems (Wang et al. (017) and drug discovery Zhou et al., 2020; Ioannidis
et al., 2020). Knowledge-gre -gaph (KG) embedding models rely on mapping the nodes and edges of the KG to a vector space by maximizing a score function for existing KG edges (Wang et al. 2017 Yang et al. 2014 Zheng et al. 2020). RGCN models (Schlichtkrull et al. 2018) have been successful in link prediction and contrary to KG embedding models can further utilize node features. The universal embeddings extracted from PanRep without labeled supervision offer a strong competitive to these supervised approaches for both node classification and link prediction tasks.
Pretraining. Pretraining models provides significant performance boost compared to traditional approaches in natural language processing (Devlin et al. 2018 Radford et al. (019) and computer vision (Donahue et al. [0144] Girshick et al., (014) Pretraining offers increased generalization capability especially when the labeled data is scarce and increased inference speed relative to end-to-
end training (Devlin et al. 2018). Parallel to work (Hu et al., 2020; Qiu et al.) 2020) proposed pretraining models for GNNs. These contemporary works supervise the models only using attribute or edge generation schemes without accounting for higher order structural similarity or maximizing the mutual information of the embeddings. Recently, (Hu et al.] 2019) introduced a framework for pretraining GNNs for graph classification. Different than (Hu et al. 2019) that focuses graph representations, PanRep aims at node prediction tasks and obtains node representations via capturing properties related to node homophily (Gleich) 2015) or node structural similarity (Rossi & Ahmed 2014). PanRep is novel pretrained model for node classification and link prediction that requires significantly less labeled points to reach the performance of its fully supervised counterparts.
3 DEFINITIONS AND PROBLEM FORMULATION
A heterogeneous graph with T node types and R relation types is defined as g {(
The node types represent the different entities and the relation types rep- resent how these entities are semantically associated each other. For example, in the IMDB network, the node types correspond to actors, directors, movies, etc., whereas the relation types correspond to directed-by by and played-in relations. The number of nodes of type is denoted by No and its associated nodal set by Vu= := {nz} { The rth relation type, &,- == {(ne, n'p) E V2 x Ve holds all interactions of a certain type among Vi and Vir and may represent that a movie is directed-by a director. Heterogenous graphs are typically used to represent KGs (Wang et al.] (017). Each node n, is also associated with an F x feature vector x This feature may be a natural language embedding of the title of a movie. The nodal features are collected in N x F matrix X. Note that certain node types may not have features and for these we use an embedding layer to represent their features.
N. The total number of nodes in G is N := Ei= N₁-
goal Unsupervised learning. Given G and X, the goal of representation learning is to estimate function g such that H == g(X,,) G). where H E - represents the node embeddings and D is the size of the embedding space. Note that in estimating g. no labeled information is available.
Universal representation learning. The universal representations H should perform well on differ- ent downstream tasks. Different node classification and link prediction tasks may arise by considering different number of training nodes and links and different label types, e.g., occupation label or education level label. Consider I downstream task, for the universal representations H it holds that
((() (H)))(()) Se, i= ie,.i,,...,,
(I)
where C(,, f()), and (1) represent the loss function, learned classifier, and training set (node labels or links) for task i, respectively and e is the largest error for all tasks. The goal of unsupervised universal representation learning is to learn H such that e is small. While learning H. PanRep does not have knowledge of (c()) f()), T(0) ).. Nevertheless, by utilizing the novel decoder scheme PanRep achieves superior performance even compared to supervised approaches across tasks.
PANREP
Our universal representation learning framework aims at embedding nodes in a low -in-mnennional space such that the representations are discriminative for node classification and link prediction.
3

---

Under review as a conference paper at ICLR 2021
Methods for learning over graphs typically rely on modeling homophily of nodes that postulates neighboring vertices to have similar attributes (Smola & Kondor 2003] Yuan et al. (014) or structural similarity among nodes (Rossi & Ahmed] 2014), where vertices involved in similar graph structural patterns possess related attributes (Donnat et al.
2018). Motivated by these methods we identify
related properties encoded in the graph data. Clustering nodes based on their attributes provides
strong signal for node homophily (Koren et al.
2009). Network motifs reveal the local structure
information for nodes in the graph (Ahmed al.
2017). Metapaths encode the heterogeneous graph
neighborhood and indicate the local connectivity (Dong et al. [017]. Finally, maximizing the mutual information among embeddings declusters node representations and provides further discriminative information (Veličković et al. 2018b).
4.1 UNIVERSAL SUPERVISION SIGNALS
In order to capture the aforementioned properties we develop four novel universal decoders. Cluster and recover supervision. Node attributes may reveal interesting properties of nodes, such as clusters of customers based on their buying power and age. This is important in recommendation systems, where traditional matrix factorization approaches (Koren al., 2009) rely on revealing clusters of similar buyers. To capitalize such information we propose to supervise the universal embeddings by such cluster representations. Specifically, we cluster the node attributes via K- means (Kanungo et al. 2002) and then design a model that decodes H to recover the original clusters. The CR-decoder modeled as a two layer MLP and is supervised by
K LCR Σ>
Crk In Crtes
where the cluster assignment Cuk is 1 if node n belongs in class and the predicted cluster assignment Önk is the output of the CR-decoder. We showcase in the appendix that this signal is superior to the attribute generation scheme used in (Hu et al.) 2020). Such supervision signal will enrich the universal embeddings H with information based on the clustering of local node features.
Motif supervision. Network motifs are sub-graphs where the nodes have specific connectivity patterns. Typical size-3 motifs for example, are the triangle and the star motifs. Each of these sub-graphs is identified by particular pattern of interactions among nodes, and reveals important properties for the participating nodes. In gene regulatory networks for example, motifs are associated with certain biological properties (Babu et al., 2004). The work in (Ahmed al., 2017) develops efficient parallel implementations for extracting network motifs. We aspire to capture structural similarity among nodes by predicting their motif information. The motivation is that nodes which might be distant in the graph may have similar structural properties as described by their motifs.
Using the method in (Ahmed al. 2017) we extract frequency vector μh, per node that shows how many times n participates to graph motifs up to size 4. This information reveals the structural role of nodes such as star-center, star-edge nodes, or bridge nodes. The motif decoder predicts this vector for all nodes using the universal representation H. This allows for information sharing among nodes which are far away the graph but have similar motif frequency vectors. The novel motif decoder modeled as two-layer MIP and is supervised by the following loss function
N
LAOO Σ
(3)
n=1
where j., is the output of the Mot-decoder for the 7ith node. Using the Mot-decoder PanRep enhances the universal embeddings by structural information encoded in the node motifs.
Metapath RW supervision. Metapaths sequences of edges of possibly different type that connect nodes in a KG. A metapath random walk (MRW) a specialized RW that follows different edge-types (Dong al. 2017).
We aspire to capture local connectivity patterns by promoting nodes participating in a MRW to have similar embeddings. Consider all node pairs for nodes rt'r) participating in MRW, the following criterion maximizes the similarity among these nodes as follows
Luww == log(1 + exp(-y x h diag(re,)) ))rr))))
where hn, and hn are the universal embeddings for nodes n, and n'e, respectively. ry is an embedding parametrized on the pair of node-types and y is if n4 and n'x e co-occur in the MRW and
4

---

Under review as a conference paper at ICLR 2021
-1 otherwise. Negative examples are generated by randomly selecting tail nodes for fixed head node with ratio 5 negatives per positive example. Link prediction is indeed a special case of the MRW supervision that considers MRWs of length 1. However, metapaths convey more information than regular links since the former can be defined to promote certain prior knowledge. For example, in predicting the movie genre in IMDB the metapath configured by the edge types (played by, played in) among node types (movie, actor, movie) will potentially connect movies with same genre and hence is desirable. The embedding per node-type pair rr, allows the MRW-dccoder to weight the similarity among node embeddings from different node types accordingly. The length of the MRW controls the radius of the graph neighborhood considered in equation and can vary from local to intermediate.
Heterogenous information maximization. The aforementioned supervision signals capture cluster- ing affinity, structural similarity and local and intermediate neighborhood of the nodes. Nevertheless, further information can be extracted by the representations by maximizing the mutual information among node representations. Such an approach for homogeneous graphs is detailed in (Veličkovid al. 2018b), where the mutual information between node representations and the global graph summaries is maximized (Hjelm et al., 2018).
Towards further refining the universal embeddings, we propose generalization of (Veličković et al. 2018b) for heterogeneous graphs. We consider a global summary vector per as == En.= h that captures the average /th node representation. We aspire to maximize the mutual information among S, and the corresponding nodes in V.. The proposed HIM decoder is supervised by the following contrastive loss function
T Σ
CHIM =
Σ 72g
log (a Ws:)) + log (1 (h Ws:)
(5)
where the bilinear scoring function (hT, Ws:) captures how close is ha, to the global summary, W a learnable matrix and h., represents the negative example used to facilitate training. Designing negative examples is a cornerstone property for training contrastive models (Veličković et al. 2018b). We generate the negative examples in by shuffling node attributes among nodes of the same type. The HIM decoder maximizes the mutual information across nodes and complements the former decoders.
Putting everything together. PanRep's final loss function is the linear combination of 5 and can be considered in the framework of deep multitask learning (Zhang al., 2014), since the GNN encoder is shared across the multiple supervision tasks and PanRep makes multiple inferences in one forward pass. Such networks are not only scalable, but the shared features within these networks induce more robust regularization and possibly boost performance (Chen et al.) (001]). Introducing adaptive weights per decoder to control its learning rate is future direction of PanRep.
4.2 PANREP ENCODER
Although the PanRep framework can utilize any GNN model as an encoder (Wu et al. 2020), in this paper PanRep uses relational (R (R)GCN encoder (Schlichtkrull al. 2018). RGCNs extend the graph convolution operation (Kipf & Welling 2017) to heterogenous graphs. The ith RGCN layer
computes the 7ith node representation h. [[+1)
as follows
h (1+1)
Σ
Σ W'ENT
!
where Ng is the neighborhood of node n under relation r, a the rectified linear unit linear function, and (!(") is a learnable matrix associated with the rth relation. Essentially, the output of the RGCN layer for node
nonlinear combination of the hidden representations of neighboring nodes weighted based on the relation type.
Alternative encoders. Several works consider designing possibly more general GNN encoders that utilize attention mechanism (Wang et al., 2019b; Veličković et al., 2018a) or graph isomorphism networks (Xu et al.] 2019). This paper proposes novel supervision signals for unsupervised learning that capture general properties of the graph data. Designing a universal encoder based on these contemporary GNN models is interesting future direction of PanRep.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Node classification results.
Train f
node2vee
meta2vee
Unsugervised
HIM
PanRep
HAN
MAGNN
Semi supervised
RGCN
PanRep-FT
40%
50.63
47.57
55.21
56.04
56.15
60.27
58.48
59.49
Mac-FI
60%
51.65
48.17
57.66
58.51
57.29
60.66
58.42
59.86
IMDB
40%
80%
51.77
51.49
48.17
55.11
57.89
55.92
60.23
57.32
60.50
58.64
58.76
61.49
59.67
Mic-FI
60%
52.79
49.87
56.57
58.41
58.42
60.88
58.55
59.75
80%
52.72
50.30
$7.79
60.14
59,24
61.33
58.89
61.39
40%
56.37
65.75
30.54
57.76
63,59
63.31
64.68
64.72
Ma-FI
60%
57.01
66.09
51.98
59.72
64.25
63.42
65.96
66.99
80%
58.05
65.75
53.25
63.03
64.37
63.89
67.67
67.90
OAG
40%
70.17
71.91
75.50
Mic-FI
60%
70.95
74.96
73.89
77.39
75.32
72.75
81:39
81.78
80%
72.24
74.73
75.31
79.76
75.24
73.45
82.38
83.17
4.3 PANREP-FT
In certain cases a very small subset of labels may be known a priori for the downstream task. In such cases it is beneficial to fine-tune PanRep's 's model to obtain refined node representations. In this context, PanRep can be considered as pretrained model and a downstream task specific loss may be applied to supervise PanRep. BERT models in natural language processing have reported state of the art results by considering such a pretrain and fine-tune framework (Devlin et al. (018)). PanRep-FT can be considered a counterpart of BERT for extracting information from heterogenous graph data. PanRep-FT combines the benefit of universal unsupervised learning and task specific information and achieves greater generalization capacity especially when labeled data are scarce (Erhan et al., 2010))
EXPERIMENTS
The proposed universal represention learning techniques are compared with state-of-the-art methods. For node classification the following contemporary methods are considered RGCN (Schlichtkrull et al. 2018), HAN (Wang et al. (2019b), MAGNN (Fu et al. 22200, node2vee (Grover & Leskoved 2016), meta2vec (Dong et al. (017) and an adaptations of the work in (Veličković et al.) | 2018b) for heterogenous graphs termed HIM. For link prediction the baseline models is RGCN (Schlichtkrull ct alaa]]2011 2018) with DistMult supervision (Yang et al.) 2014) that uses the same encoder as PanRep. The Mot-decoder and RC-decoder employ a 2-llayer MLP. For the MRW-decoder use length-2 MRWs. The parameters for all methods considered are optimized using the performance on the validation set.
Datasets. We consider subset of IMDB dataset (IMD 2020) containing 11,616 nodes with 3 node- types and 17,106 edges from 6 edge-types. Each movie is associated with a label representing its genre and with feature vector capturing its keywords. We also subset of the OAG dataset (OAG) 2020) with 23,696 with 4 node types (authors, affiliations, papers, venues) and 90,183 edges from 14 edge-types. In OAG we did not use MOT supervision since (Ahmed et al. 2017) is not applicable. Each paper is associated with a label denoting the scientific area and with an embedding of the papers' text. Finally, we utilize the DRKG constructed in (Toannidis et al. 2020). DRKG contains 5,874,261 biological interactions belonging to 107 edge-types among 97,238 biological entities from 13 entity-types. For further details on the datasets and configuration of methods see the appendix.
Scalability. PanRep is implemented using Deep Graph Library (DGL)'s mini-batch training, which scalable w.r.t. time and space. During mini-batch training subset of seed nodes is sampled along with the K-hop neighbors of the seed nodes, where K is the depth of the GNN. Furthermore, the number of edges each level is bounded by using neighbor sampling. This allows training on graphs with billions of nodes/edges since only a fraction of the nodes/edges will be loaded on the GPU per mini-batch. PanRep's 's complexity controlled by the RGCN encoder whose computation complexity of
is ((FFK) for sparse graphs, where B is the batch size, K is the number of layers and F is the maximum number of neighbors per node. By using neighbor sampling, F' is usually less than 10-20 and in our experiments we found that K=2 often works well. The decoders' complexity is significantly lower and do not add but a constant factor to the overal complexity and hence the overall per-epoch runtime is similar to that of RGCN.
5.1 NODE CLASSIFICATION
In this experiment we compare supervised and unsupervised methods for classification. First, the labeled nodes split in 10% training, 5% validation, and 85% testing sets. The supervised methods utilize the training labels whereas the unsupervised methods do not use them in calculating the node embeddings. Then the embeddings corresponding to the 85% testing nodes are further split to training
6

---

Under review as a conference paper at ICLR 2021
OMG
ONG
Pankep FT a Ponitep + RGON
Panieep ITT an Pankep ROCN
Pankeep-FT Pankepp RGON
Pankep PT Panktep RGCN
100
70-
90-
MRR
Hi-10
60-
H8-10
85
50-
80
60
70
80
60
70
80
50
60
70
90
00
70
80
90
Traming link percensage
Training percentage
Training link percenaaee
Training link
Figure 2: MRR and Hit-10 for link prediction across different percentages of testing links.
Table 2: Node classification results for different labeled supervision splits.
embeeklings
10%
20%
Dalasets
Metrics
Train:
FanRep
ROCN
PanRop-F1
ROCN
PanRep-F1
ROCN
Pankee-F" -
36.04
35.12
39.49
Macro-F1
60%
58.51
55.20
$9.39
58.42
59.86
60.98
62.91
KI%
60.23
55.55
61.27
58.76
61.49
61.10
62.72
40%
53.92
35.27
56.92
38.64
39.67
61.49
63.17
Micro-F1
60%
38:4
55.39
59.45
58.55
59.75
61.17
62.89
60.14
55.62
40%
57.76
35.31
64.99
64.68
64.72
67.07
63.31
Macro-F1
59.72
55.99
66.62
65.96
66.99
67.58
66,25
OAG
80%
40%
63.03
75.50
56.36
78.00
80.19
68.94
81.92
66.10
80.36
68.60
82,57
67.67
81.17
67.90
Micro-F1
77.39
28.07
81.39
80%
79.76
78.44
82,52
82,38
83.17
82,30
82,31
and testing sets and linear SVM is trained. This evaluation setting allows us to directly compare the different supervised and unsupervised approaches.
We report the Macro and Micro Fl accuracy for different training percentages of the 85% nodes fed to the SVM classifier in Table II It is observed that PanRep outperforms significantly other unsupervised approaches as well as some supervised approaches. supervised counterpart RGCN that uses node labels for supervision. Metapath2vec (Dong et al.) 2017) reports competitive performance for OAG in Macro-FI score but underperformed in Micro-Fl. Nevertheless, in this experiment Metapath2vec only uses the best performing metapath that is paper- venue-paper, which is considerably better than that of most other metapaths. This way of selecting the metapath, gives Meta2vec (unfair) advantage over PanRep, which uses all metapaths of length as it computes a universal representation (along with the other supervision signals). PanRep-FT outperform outperforms significantly RGCN that uses the same encoder, which is testament to the power of pretraining models. Finally, PanRep-FT matches and outperforms in certain splits the state-of-the-an MAGNN that uses more expressive encoder. PanRep's universal decoders enhance the embeddings with additional discriminative power that results to improved performance in the downstream tasks.
certain splits, PanRep outperforms its
reports the accuracy of the PanRep-FT and the encoder RGCN, which is trained directly for the semi-supervised learning task to obtain the embeddings. PanRep-FT consistently outperforms RGCN across most SVM splits, whereas PanRep-FTTs advantage over RGCN decreases more training data are available, which is justifiable since more labels diminish the advantage of pretraining. Moreover, PanRep-FT performs only 100 finetuning epochs with labeled supervision, which is significantly less to the 800 epochs of labeled supervision by RGCN. Nevertheless, PanRep outperforms the supervised RGCN embeddings for 5% training labels. This demonstrates the importance of using PanRep as pretraining method. RGCN reports similar performance across SVM splits, whereas PanRep-FT -T increases with more supervision. These results suggest that PanRep-FT'ss embeddings have higher generalization capacity.
Table reports ablation study by using different decoder subsets. PanRep that uses all signals obtains the best performance. The decoders HIM and MRW and the their combination exhibit the second best performance. Nevertheless, the full supervision in PanRep leads to the superior performance. Section B in the appendix includes additional experiments.
5.2 LINK PREDICTION
Our universal embedding framework is further evaluated for link prediction using the IMDB and OAG datasets. The MRW decoder is used to evaluate the performance of PanRep in link prediction. Figure reports the MRR, and Hit- 10 scores of the baseline methods along with the PanRep and PanRep-FT methods. We report the performance of the methods for different percentages of links used for training. Observe that PanRep-FT consistently outperforms the competing methods and
7

---

Under review as a conference paper at ICLR 2021
Table 3: Ablation study for different supervision signals.
Darasets
Metries
Train %
PanRop
40%
55.21
54.54
55.32
42.28
32.21
56.04
Macro-F1
57.66
56.12
57.24
43.41
35.16
58.51
806
57.89
56.64
57.74
44.31
35.66
60.23
IMDB
40%
55.11
54.36
55.53
43.66
39.13
55,92
Micro-FI
60%
56.57
55.91
56.25
40.03
SIP
57.79
56.49
58.42
45.65
40.66
60.14
55.92
15:46
Macro-FI
6I8S
51.98
58:40
58.91
15.48
59.72
OAG
SIPS
53.25
60.61
61.74
15:54
63.03
40%
71.91
74.39
74.05
63.06
75.50
Micro-F1
69%
73.89
76.76
76.33
63.14
77.39
75.31
Table 4: Drug inhibits gene scores for Covid-19,
Drug PanRep-FT FT
RGCN
hits
Drug
# hits
Drug
# hits
Drug
hits
Lesartan
Thalidomide
41
Chloraquine
Tofacitinib
Chlomquine
198
Hydrox
19
Golchicine
41
Rihavirin
32
Deferoxamine
Tetrandrine
13
Tetrandrine
Methylpreednisoooe
Ribavirin
101
Bculizumab
10
Oscltamivir
37
Deferoxamine
30
Methhy|prednisolooee
Tocilizumah
9
Azithromycin 36
Thalidomide
We retain the top- 10 drugs based on their number of hiss for each method. Not that random classifier will result to appeuximaelly 5.3 per drug. This suggests that the reported predistions are significantly betier than random.
the performance gain increases as the percentage of training links decreases. This corroborates the advantage of pretraining GNNs for link prediction. Note that PanRep reports similar performance with RGCN that is trained solely in link prediction. This result confirms the success of the universal embeddings in link prediction.
5.3 DRUG REPURPOSING
Drugrropug aims discovering the most effective existing drugs to treat a certain discase. Using the Drug Repurposing Knowledge Graph (DRKG) (Ioannidis et al. 2020), we compare the drug repurposing results in C9vid-19 among PanRep-FT that is finetuned in link prediction and the baseline RGCN. We employ L = 1 hidden layer with D = 600 and train for 800 epochs both networks. Drug-repurposing can be formulated as predicting direct links in the DRKG. Here, we attempt to predict whether a drug inhibits a certain gene, which is related to the target disease. We identify 442 genes that are related with the Covid-19 disease. We select 8, 104 FDA-approved drugs in the DRKG as candidates; see also the appendix for further information in this experiment. To validate our predictions we use 32 Covid- 19 clinical trial drugs. For each gene node we calculate with RGCN and PanRep-FT an inhibit link score associated with every drug. Next, we score all mong-inhibits-gen triples and rank them per target gene. We obtain in this way 442 ranked lists of drugs, one per gene node. Finally, to assess whether our prediction is in par with the drugs used for treatment, we check the overlap among the top 100 predicted drugs and the drugs used clinical trials per gene. Table lists the clinical drugs included in the top- 100 predicted drugs across all the genes with their corresponding number of hits for the RGCN and PanRep-FT. It can be observed, that several of the widely used drugs in clinical trials appear high the predicted list in both prediction. Furthermore, PanRep-FT reports a higher hit rate than RGCN, which corroborates the benefit of using the universal pretraining decoders. The universal representation endows PanRep with increased generalization power that allows for accurate link prediction performance when training data are extremely scarce as is the case of Covid-19. While this study, does not recommend specific drugs, it demonstrates powerful deep learning methodology to prioritize existing drugs for further investigation, which holds the potential of accelerating therapeutic development for Covid-19.
4
CONCLUSION
This paper develops a novel framework for unsupervised learning of universal node representations on heterogenous graphs termed. Pankep supervises the GNN encoder by decoders attuned to model the clustering of local node features, structural similarity among nodes, the local and intermediate neighborhood structure, and the mutual information among same-type nodes. To further facilitate cases where limited labels are available we implement PanRep-FT. Experiments in node classification and link prediction corroborate the competitive performance of the learned universal node representa- tions compared to unsupervised and supervised methods. Experiments on the DRKG showcase the advantage of the universal embeddings in drug repurposing.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
www. imdb............ com, 2020.
www www.ooennacdemmc..a.oooo 2020.
Amr Ahmed, Nino Shervashidze, Shravan Narayanamurthy, Vanja Josifovski, and Alexander J Smola. Distributed large-scale natural graph factorization. Proceedings of the 22nd international conference on World Wide Web, pp. 37-48, 2013.
Nesreen K Ahmed, Jennifer Neville, Ryan A Rossi, Nick G Duffield, and Theodore I. Willke. Graphlet decomposition: Framework, algorithms, and applications. Knowledge and Information Systems, 50(3):689-722 2017.
M Madan Babu, Nicholas M Luscombe, L Aravind, Mark Gerstein, and Sarah A Teichmann. Structure and evolution of transcriptional regulatory networks. Current opinion structural biology. 14(3): 283-291, 2004.
M. Belkin, I. Matveeva, and P. Niyogi. Regularization and semi-supervised learning on large graphs. In Proc. Annual Conf. Learning Theory, volume 3120, pp. 624-638, Banff, Canada, Jul. 2004. Springer.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs/1206.5538, 1:2012, 2012.
Shaosheng Cao, Wei Lu, and Qiongkai Xu. Grarep: Learning graph representations with global structural information. In Proceedings of the 24th ACM international on conference on information and knowledge management, pp. 891-900, 2015.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient nor- malization for adaptive loss balancing in deep multitask networks. arXiv preprint 1rX1v:1711.00255, 2017.
Jacob Devlin, Ming-Wat Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647-655, 2014.
Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 135-144, 2017.
Claire Donnat, Marinka Zitikk, David Hallac, and Jure Leskovec. Learning structural node embed- dings via diffusion wavelets. In Proc. Intl. Conf. on Knowledge Disc. and Data Mining (KDD), 2018.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep lcarning? Journal of Machine Learning Research, 11(Feb).625-660, 2010.
Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference 2020, pp- 2331-2341 2020.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-588533 587,-587-10 2014.
David F Gleich. Pagerank beyond the web. SIAM Review, 57(3):321-363, 2015.
David E Gordon, Gwendolyn M Jang, Mehdi Bouhaddou, Jiewei Xu, Kirsten Obernier, Matthew O' meara, Jeffrey Z Guo, Danielle L Swaney, Tia A Tummino, Ruth Huttenhain, et al. A sars- cov-2-human protein-protein interaction map reveals drug targets and potential rruuroopu....ing. Nature, 2020. doi: 10. 1038//41586-0200 588-000-2286-999

---

Under review as a conference paper at ICLR 2021
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SICKDD international conference on Knowledge discovery and data mining, pp- 855-864, 2016.
William Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint
2017.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 110iv:8808.0666 2018.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2019.
Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai- Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In arxiv, 2020.
Vassilis N. loannidis, Xiang Song, Saurav Manchanda, Mufei Li, Xiaoqin Pan, Da Zheng, Xia Ning, Xiangxiang Zeng, and George Karypis. Drkg drug repurposing knowledge graph for covid-19 19. com/ wtt.e/omm///c//eeee Baaanddd/DRKGG 2020.
https://gitt
Tapas Kanungo, David M Mount, Nathan $ Netanyahu, Christine D Piatko, Ruth Silverman, and Angela Y Wu. An efficient k-means clustering algorithm: Analysis and implementation. IEEE transactions on pattern analysis and machine intelligence, 24(7):881-892, 2002.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In Proc. Int. Conf. on Learn. Represantions, San Diego, CA, USA, May 2015.
Thomas N Kipf and Max Welling. Supi-supe classification with graph convolutional networks. In Proc. Int. Conf. on Learn. Represantions, Toulon, France, Apr. 2017.
Kyle Kloster and David F Gleich. Heat kernel based community detection. In Proc. Intl. Conf. Knowledge Disc. and Data Mining (KDD), pp. 1386-1395, 2014.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu. Asymmetric transitivity preserv- ing graph embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1105-1114, 2016.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 701-710, 2014.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang. Ming Ding, Kuansan Wang. and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. In arxiv, 2020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Ryan A Rossi and Nesreen Ahmed, Role discovery in networks. IEEE Transactions on Knowledge and Data Engineering, 27(4):1112-111 2014.
Michael Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pp. 593-607. Springer, 2018.
Jingbo Shang, Meng Qu, Jialu Liu, Lance M Kaplan, Jiawei Han, and Jian Peng. Meta-path guided embedding for similarity search in large-scale heterogeneous information networks. arXiv preprint arXiv: 160.00660 2016.
10

---

Under review as a conference paper at ICLR 2021
Chuan Shi, Binbin Hu, Wayne Xin Zhao, and S Yu Philip. Heterogeneous information network embedding for recommendation. IEEE Transactions on Knowledge and Data Engineering, 31(2): 357-370, 2018.
A. Smola and R. I. Kondor. Kernels and regularization on graphs. Learning Theory and Kernel Machines, pp. 144 158. Springer, 2003.
Johan AK Suykens and Joos Vandewalle. Least squares support vector machine classifiers. Neural processing letters, 9(3):293-300, 1999.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In Proceedings of the 24th international conference world wide web, pp. 1067-1077, 2015.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In Proc. Int. Conf. on Learn. Represantions, 2018a.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv:1809.1034/ 2018b.
Minjie Wang, Lingfn Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye. Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et Deep graph library: Towards efficient and scalable deep learning on graphs. arXiv 2019a.
preprint 19x9::9909.011
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12): 2724-2743 2017.
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In The World Wide Web Conference, pp. 2022 2022-2032, 2019b.
Xindong Wu, Xingquan Zhu, Gong-Qing Wu, and Wei Ding. Data mining with big data. IEEE transactions on knowledge and data engineering, 26(1):97-177, 2013.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In Proc. Int. Conf. on Learn. Represantions, 2019.
Bishan Yang, Wen-tau Yih, Xiaodong Hc, Jianfeng Gao, and Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv: 112.4:1412.6575, 2014.
Guangchao Yuan, Pradeep Murukannaiah, Zhe Zhang, and Munindar Singh. Exploiting sentiment homophily for link prediction. In Proceedings of the 8th ACM Conference on Recommender systems, pp. 17 17-24, 2014.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference on computer vision, pp. 108 Springer, 2014.
Da Zheng, Xiang Song, Chao Ma, Zeyuan Tan, Zihao Ye, Jin Dong, Hao Xiong, Zheng Zhang, and George Karypis. Dgl-ke: Training knowledge graph embeddings at scale. arXiv preprint arXiv:2004.08532. 2020.
Yadi Zhou, Yuan Hou, Jiayu Shen, Yin Huang, William Martin, and Feixiong Cheng. Network-based drug repurposing for novel coronavirus 2019-ncov/sars-cov Cell discovery, 6(1):1-18, 2020.

---

Under review as a conference paper at ICLR 2021
Table 5: Attribute generation supervision against clustering and recover supervision for different number of clusters K in the node classification task.
Datasets
Metries
Train ATTRIBUTE GENERATION
CR
4
CR K
10
40%
17.52
29.52
31,94
32.21
31.22
Macro-FI
60%
17.53
30.75
34.50
35.16
34.19
80%
17.30
3145
35.02
35,66
34.82
IMDB
40%
35.66
38.51
39.40
39.13
38.38
Micro-F1
60%
35.70
38.83
41.02
41.13
40.03
80%
35.07
39.54
41.16
41.39
40.66
A IMPLEMENTATION FRAMEWORK
The methods presented in this paper are implemented in the efficient deep graph learning (DGL.
library Wang et al. (2019a). PanRep is implemented using the mini-batch training framework that facilitates training for very large graphs even with limited computational resources The competing methods RGCN, MAGNN and HAN are also implemented using the DGL. PanRep experiments are executed on AWS P3. P instances with 8 GPUs each having 16GB of memory.
B CLUSTER AND RECOVER COMPARED TO NODE ATTRIBUTE GENERATION
In this section we compare the cluster and recover supervsion signal to the attribute generation supervision that is employed by several GNN pretraining methods. In this experiment employ the setting of Section |.1 for node classification. The attribute generation supervision employs a 2-layer MLP network that attempts to reconstruct the original nodal attributes given the node embeddings. In Table 5 we report the node classification accuracy for different number of clusters considered in the CR supervision. It can be observed that the CR methods significantly outperform the attribute generation method. One reason for this is that the attributes are already considered by the GNN encoder in calculating the embeddings, and this supervision does not increase the predictive capacity of the model. On the other hand the clustering supervision allows the GNN to discover clusters of nodes with similar attributes and hence refines the learned embeddings.
C METHODS
Different competing methods include RGCN Schlichtkrul et al. (0188), HAN Wang et al. (219b)),
MAGNN Fu et al. (2020), node2vec Grover & Leskovee (2016), meta2vec Dong et al. (2017) and an adaptation of the work in Veličković et al. (2018b) for heterogenous graphs termed HIM. For link prediction the baseline model is RGCN Schlichtkrull al.) (2018) with DistMult supervision Yang et al. (2014) that uses the same encoder as PanRep. The Mot-decoder and RC-decoder employ 2-layer MLP. For the MRW-decoder considered are optimized using the performance on the validation set. For the majority of the experiments PanRep uses a hidden dimension of 300, 1 hidden layer, 800 epochs of model training, 100 epochs for finetuning, and an ADAM optinzer/Knggma & Ba (2015) with learning rate of 0.001. For link prediction finetuning PanRep uses a DistMult model Yang et (2014) whereas for node classification it uses logistic loss.
use length-2 MRWs. The parameters for all methods
D DRUG REPURPOSING EVALUATION
Our work focused entirely on evaluating the ranked list of candidate drugs against of 32 drugs that, at the time of submission, were in Covid-19 clinical trials (Gordon et al., 2020) Zhou et al.) 2020). Specifically, given a gene whose associated protein is potential drug target, we used the approach described in section 5.3 to rank the 8104 FDA approved drugs and select the 100 highest ranked drugs. We evaluated the performance of method by looking 100 highest ranked drugs and the 32 drugs in Covid-19 19 trials. In our experiments, we used 442 genes that researchers have identified as potential targets for Covid-19 and computed 442 ranked lists and associated intersections. We aggregated the results by counting the number of times each one of the 32 drugs were in these intersections, and the drugs with the highest counts are shown in Table 4.
intersection between these
https://www.dal. com
12

---

Under review as a conference paper at ICLR 2021
Table 6: Statistics of datasets.
Dataset
Node
Edge
movie (M): 4,278
M-directed by-D: 4,278, D-directed-Mr 4,278
IMDb
director (D); 2,081
# M-played by-A: 12,828, A-played-M: 12,828
actor (A): 5,257
# P-in journal-V; 3941, V-joumal has-P: 3941
# P-conference-VV V: 3368, V-conference has-P: 3368
# author (A): 13,720
P-cites-P: 3327, P-cited by-P: 3327
# paper (P): 7,326
OAG
A-writes last-P: 4522, P-written by last-A: 4522
affiliation (Af): 2,290
A-writes as other-P: 7769, P-written by other-A: 7769
# venue (V): 782
# A-writes as first-P: 4795, P-written by first-A: 4795
# A-affiliated with-Af: 17035, Af-affiliated with-A: 17035
Table 7: Number of nodes per node type in the DRKG and the data sources.
Entity type
Drugbank
GNBR
Hetionet
STRING
IntAct
DGIdb
Bibliography
Total Entities
Anatomy
400
400
Atc
4,048
4, 048
Biological Process
381
11, 381
Cellular Component
1,391
1, 391
Compound
9, 708
161
1,538
153
6, 0,3
6, 250
24, 313
Disease
1,182
4. 746
257
103
Gene
4,973
27, 111
14,
18, 316
321
2,551
181
39, 220
Molecular Function
8,884
884
Pathway
1, 822
1, 822
Pharmacologic Class
345
345
Side Effect
701
6,
Symptom
415
Tax
215
215
Total
19,
43,033
45, 279
18,316
8,899
9.464
238
E DATASETS
E. DRKG
The Drug Repurposing Knowledge Graph (DRKG) contains 97055 entities belonging to 13 entity- types Joannidis al. (2020). The type-wise distribution of the entities is shown Table DRKG contains total of 5869294 triplets belonging to 107 edge-types. Table the number of triplets between different entity-type pairs for DRKG and various data sources. The DRKG publicly available
E.2 IMDB AND OAG
IDMB IMD (2020) is movie database including information about the cast, production crew, and plot summaries. A subset of IMDb used after data preprocessing Table Movies are labeled as one of three classes (Action, Comedy, and Drama) based on their genre information. Each movie is also described by bag-of-wordd representation its plot keywords.
OAG (2020) is bibliography website. We preprocess the data and retain the subgraph in Table The papers are divided into research areas. Each paper described by BERT embedding of paper's s title.
13

---

Under review as a conference paper at ICLR 2021
Table 8: Number of interactions in the DRKG and the data sources.
Entety-type pair
Drugbank
GNBR
Hetionet
STRING
IntAct
DGIdb Bibliography
Total interactions
('Gene'. 'Gene") ('Compound', 'Gene") (DDiccase.. 'Gene") ('Ate,, Compound")
667, 22
474,520 526
496 708
211,316
58. 029
2,350, 931
24, 801
80, 803
51, 429
1,805
26, 290
25, 666
210, 794
35,
27,977
461
123, 837
15. 750
15, 750
('Compound', "Compound")
1,379, 271
486
1. 385, 757
("Compound,, "Disease')
4, 1,968
145
895
('Gene', 'Tax')
14, 663
14, 663
"Biological Process'. 'Gene')
559, 504
559, 504
(DDisease,, SSmppom')
357
3, 357
(' ('Anatomy', DDscaase)
3, 602
3, 606
(Diseass', DDisease'
543
543
(' l'Anatomy'. 'Gene")
726, 195
726, 495
('Gene', 'Molecular Function")
97, 222
97, 222
('Compound', PPharmacooogic Class')
029
1. 029
('Cellular Component", 'Gene")
73. 566
73, 566
GeGe' : 'Pathway')
84, 372
84, 372
('Compound,, Effect')
138, 911
138, 911
Total
1,424, 790
335,309
2, 197
496,
256, 20,290
756
874, 261
14


Reviews:
[{"review_id": "e1x_PnwPUzX", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n(1) This paper is well-organized and easy to follow, clearly presenting the composition of the proposed model.\n\n(2) Motivation of this paper is strong, and readers may foresee rich applications.\n\n(3) Each model component is with a clear goal, facilitating the interpretation of the proposed approach.\n\nCons:\n\n(1) This paper is not technical. All techniques are not originally proposed, and the whole model is simply ‘putting everything together’.\n\n(2) For all ablation studies, no evidence is shown which component contributes to the performance improvement compared with the best single-component performance.\n\n(3) No performance variance is provided to make the improvement more convincing considering the close performances of some cases.\n\n- - - -\n\nDetailed Comments:\n\nThis paper proposes an approach for unsupervised learning of universal node representations on heterogenous graphs. Specifically, four components are presented with individual loss functions to ensure the representations encoding a certain category of information, and the final model is attained by ‘putting everything together’. Overall, this paper is well-organized and easy to follow, clearly presenting a well-motivated model and how each component handles the information extraction. We can also foresee great application value with the improved fitting ability on universal node representations. However, all techniques are not originally proposed, making this paper less technical. Besides, no evidence is shown which component contributes to the performance improvement compared with the best single-component performance in the ablation study. Also, no performance variance is provided to make the improvement more convincing considering the close performances of some cases in the experiments. To make the paper more convincing, I would have the following suggestions:\n\n(1) Ablation study should be enriched. For the current ablation study, only one component contributes to the main performance. Both the specification of where the performance improvement comes from and how components cooperation facilitates the performance are of little evidence.\n\n(2) More details should be provided on why larger training datasets (e.g., from 40% to 80%) can degrade the performance of some baselines.\n\n(3) For experiments, variance or standard deviation should be provided to make the minor improvements in some cases convincing. Also, performance on more diverse datasets should also be presented for generalization purposes.\n\n(4) In drug-repurposing experiments, the conclusion can also be obtained from Table 4 that PanRep-FT tends to have long-tail effect; however RGCN shows more stable performance. To better evaluate the performance, it is better to present the distribution among all drugs and evaluate the performance of the two models from different perspectives including application consideration.\n\n(5) There is no information on how to balance different loss terms. For example, at least five loss terms should be delicately tuned for a balance in PanRep-FT. However, the author does not present how to balance the weights among loss terms. Besides, I would argue the experimental weights among loss terms should be presented in detail to determine how each component contributes to the final performance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qq5ubFiM5KA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed introduces a problem formulation of universal unsupervised learning. They develop an unsupervised node representation learning method by combining four signals: (1) cluster and recover supervision, (2) motif supervision, (3) metapath random walk supervision, and (4) heterogeneous information maximization. Aside from conducting experiments on node classification and link prediction, they apply their method on drop-repurposing knowledge graph to discover drugs for Covid-19.\n\nI agree that there is contribution to putting these supervision signals together and showing results on drug-repurposing for COVID-19. However, the proposed approach is rather a mix of previous works and hence not novel.\n- All four supervision signals are based on existing works. What's the novelty in all four signals? Recovering cluster assignments, calculating motif frequency vectors, and proximity-based embedding (i.e. RW) are all well-known methods in the graph domain and the objective used in \"Heterogenous information maximization\" to me seems nearly identical to the one proposed in Deep Graph Infomax [1]  \n- The performance improvement can simply a result of loss ensembles. How does each individual supervision signal contribute to the performance? An ablation study discussing that should be conducted.\n\nFor the cluster and recover supervision signal, Panrep attempts to recover results obtained from K-Means. However, \n- K-Means perform poorly on many datasets (e.g. graphs where true communities are highly overlapping). I doubt that learning to recover such signals would be beneficial.  \n- how do you decide on the number of communities $K$ and does it affect performance a lot? \n- Equation (2)  should be permutation invariant to communities (invariant to the ordering of communities)\n\n[1] Petar Velickovi ˇ c, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon ´\nHjelm. Deep graph infomax. arXiv preprint arXiv:1809.10341, 2018b.", "extra_scores": {}, "extra_text": {}}, {"review_id": "J_uYjcV5s8r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a universal and unsupervised GNN-based representation learning (node embedding pretraining) model named PanRep for heterogeneous graphs, which benefits a variety of downstream tasks such as node classification and link prediction. More specifically, employing an encoder similar to R-GCN, PanRep utilizes four different types of universal supervision signals for heterogeneous graphs, i.e. cluster and recover, motif, metapath random walk and heterogeneous info maximization, to better characterize the graph structures. PanRep can be further fine-tuned in a semi-supervised manner with limited labeled data, known as PanRep-FT, for specific applications. Experiments on benchmark datasets have shown the effectiveness and performance gain over other unsupervised and some supervised baseline approaches. One example use case is applied to COVID-19 drug repurposing to identify potential treatment candidates. \n\nStrong aspects: \n(1) The motivation, problem formulation, and model illustration are well explained. \n(2) Proposed four types of universal supervision signals are interesting and generalizable for all (heterogeneous) graphs.\n(3) Extensive experiments, ablation study, and case study are supportive. \n\nWeak aspects: \n(1) Some comparable SOTA baselines may not be considered. \n(2) The heterogeneous information maximization signal seems not clearly explained. (See detailed comments below)\n\nDetailed comments: \n(1) It is strongly recommended in related work to explicitly explain the difference from other graph pretraining models such as [1,2] and the reason why they are excluded from baselines. Though some work (partially) focus on graph-level application, some node-level pretraining methods are generally comparable and applicable for heterogeneous graph representation learning. \n\n(2) Regarding the drug repurposing task, some baseline approaches (TransE, etc) implemented by DLG-KE on DKRG dataset are not reported [4]. It would be more beneficial and convincing to include the results as well by comparing the #hit other than R-GCN. \n(3) HIM is an important component from the observations in the ablation study, it requires a clearer explanation for better \nunderstanding. Some questions are: For the bilinear scoring function, why one node close to its global summary is the ultimate goal for HIM while other signals are used to learn its structures and the nodes are different from each other even though they are of the same type? Why is it reasonable to design contrastive negative samples as introduced in PanRep? What is the type of information of the nodes are fully not available (for example, the node type is not defined in the ontology)? \n\nReferences:\n[1] Hu, Weihua, et al. \"Strategies for Pre-training Graph Neural Networks.\" ICLR (2020). \n\n[2] Hu, Ziniu, et al. \"Heterogeneous graph transformer.\" Proceedings of The Web Conference 2020. 2020.\n\n[3] Ioannidis, Vassilis N., et al. \"Drkg-drug repurposing knowledge graph for covid-19.\" (2020). Github: https://github.com/gnn4dr/DRKG/tree/master/drug_repurpose  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DUqs_VHqULL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "PanRep: Universal node embeddings for heterogeneous graphs\n\nThe manuscript proposes PanRep, an unsupervised node embedding learning model for heterogeneous graphs. PanRep considers four unsupervised tasks of cluster, motif, heterogenous structure, and mutual information (contrastive learning). Experiments on two different tasks (i.e., node classification and link prediction) demonstrate that PanRep outperforms some baseline methods. \n\nPros\n1. The problem is interesting and important. It is meaningful to study heterogeneous graph pre-training. \n2. The presentation is overall good. The content is clear. \n3. Four interesting unsupervised tasks are proposed. \n\nCons\n1. The novelty of PanRep is incremental. There are a number of studies [1,2,3,4,5] for graph pre-training. Despite the difference in unsupervised task design, I did not find exciting thing from this work. \n\n2. Experiments should be improved. Baseline methods are relatively weak, it is better to compare and discuss more recent work for graph pre-training [1,2,3,4]. \n\n[1] Strategies for Pre-training Graph Neural Networks, ICLR 2020\n\n[2] GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training, KDD 2020\n\n[3] Gpt-gnn: Generative pre-training of graph neural networks, KDD 2020\n\n[4] Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs, NeurIPS 2020\n\n[5] Self-supervised Learning on Graphs: Deep Insights and New Directions, ArXiv\n\nTo summarize, the novelty of this work is incremental and the experiments could be improved. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the paper is clearly written overall and well motivated, reviewers raised several crucial concerns and, unfortunately, the authors did not respond to reviews.

During the discussion, reviewers agree with that this submission is not ready for publication. In particular, empirical evaluation is not thorough as important baselines are not included and discussion is not convincing.

I will therefore reject the paper.
For future submission, I strongly recommend the authors to do author response. There are many cases where the reviewers change their scores based on the interaction between the authors and the reviewers, which is healthy for the review process.
Decision: Reject