Zero-shot Transfer Learning within a Heterogeneous Graph via Knowledge Transfer Networks
Authors: Minji Yoon
John Palowitch
Dustin Zelle
Ziniu Hu
Russ Salakhutdinov
Bryan Perozzi
Venue: NeurIPS 2022 Accept (2022)
Keywords: graph neural networks
heterogeneous graph
heterogeneous graph neural networks
transfer learning
domain adaptation

Abstract: Data continuously emitted from industrial ecosystems such as social or e-commerce platforms are commonly represented as heterogeneous graphs (HG) composed of multiple node/edge types. State-of-the-art graph learning methods for HGs known as heterogeneous graph neural networks (HGNNs) are applied to learn deep context-informed node representations. However, many HG datasets from industrial applications suffer from label imbalance between node types. As there is no direct way to learn using labels rooted at different node types, HGNNs have been applied to only a few node types with abundant labels. We propose a zero-shot transfer learning module for HGNNs called a Knowledge Transfer Network (KTN) that transfers knowledge from label-abundant node types to zero-labeled node types through rich relational information given in the HG. KTN is derived from the theoretical relationship, which we introduce in this work, between distinct feature extractors for each node type given in an HGNN model. KTN improves the performance of 6 different types of HGNN models by up to 960% for inference on zero-labeled node types and outperforms state-of-the-art transfer learning baselines by up to 73% across 18 different transfer learning tasks on HGs.

Zero-shot Transfer Learning within a Heterogeneous Graph via Knowledge Transfer Networks
Minji Yoon" Carnegie Mellon University
John Palowitch Google Research
Dustin Zelle Google Research
Ziniu Hu" University of California Los Angeles
Ruslan Salakhutdinov Camegic Mellon University
Bryan Perozzi
Google Research
Abstract
Data continuously emitted from industrial ecosystems such as social e-commerce platforms are commonly represented as heterogeneous graphs (HG) composed of multiple node/edge types. State-of-the-art graph learning methods for HGs known as heterogeneous graph neural networks (HGNNs) are applied to learn deep context-informed node representations. However, many HG datasets from industrial applications suffer from label imbalance between node types. As there is no direct way learn using labels rooted at different node types, HGNNs have been applied on only a few node types with abundant labels. We propose a zero- shot transfer learning module for HGNNs called Knowledge Transfer Network (KTN) that transfers knowledge from aabel-abundant node types to zero-labeled node types through rich relational information given in the HG. KTN is derived from the theoretical relationship, which we introduce in this work, between distinct feature extractors for each node types given in HGNN model. KTN improves performance of 6 different types of HGNN models by up to 960% for inference on zero-labeled node types and outperforms state-of-the-arr transfer learning baselines by up 73% across 18 different transfer learning tasks on HGs.
1 Introduction
Large technology companies commonly maintain large relational datasets, derived from their internal logs, that can represented or joined into massive heterogeneous graph (HG) composed of nodes and edges with multiple types (30). For instance, in e-commerce networks, there are product, user, and review nodes, all interconnected by many edge types that represent forms of interactions such as spending (user-product), reviewing (user-review), and reviews-of (product-review)) To learn powerful features representing the complex multimodal structure of HGs, various heterogencous
graph neural networks (HGNN) have been proposed (15 26 35 43).
A common issue in these industrial applications of HGNNs is the label imbalance among different node types. For instance, publicly available content nodes such as those representing video, text, and image content are abundantly labelled, whereas labels for other types (such user account nodes) may be much more expensive to collect (or even not available, e.g. due to privacy restrictions). This means that in most standard training settings, HGNN models can only learn make good inferences for a few label-abundant node types, and can usually not make any inferences for the remaining node types, given the absence of any labels for them.
If there is pair of label-abundant and zero-labeled node types which share an inference task, could we transfer knowledge between them? One body of work has focused transferring knowledge between
Work done while interning at Google
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

nodes of the same type from two different HGs (i.e., graph-to-graie transfer learning) (16) 40). However, these approaches are not applicable in many real-world scenarios for three reasons. First. any external large-scale HG that could be used in graph-to-graph transfer learning setting would almost surely be proprietary. Second, even if practitioners could obtain access to an external industrial HG, it is unlikely the distribution of that (source) graph would match their target graph well enough to apply transfer learning. Finally, node types suffering label scarcity are likely to suffer the same issue on other HGs (e.g. user nodes).
In this paper, we introduce a zero-shot transfer learning approach for a single HG (assumed to be fully-owned by the practitioners). transferring knowledge from labelled to unlabelled node types. This setting is distinct from any graph-to-gren to-grohhgghhho transfer learning scenarios, since the source and target domains exist in the same HG dataset, and are assumed to have different node types. Our model utilizes the shared context between source and target node types; for instance, in the e-commerce network, the latent (unknown) labels of user nodes can be strongly correlated with spending/reieewigg patterns that are encoded in the cross-edges between user nodes and product/review nodes. We propose novel zero-shot transfer learning problem for this HG learning setting follows:
Informal Problem Definition 1. Zero-shot cross-type transfer learning running on a HG:
Given heterogeneous graph G with node types {s, t....) }}- with abundant labels for source type but no labels for target rype t, can we train HGNNs to infer the labels of target-type nodes?
A naïve solution to this problem would be re-use an HGNN pre-trained the source nodes for target node inference, given that both domains exist in the same HG. However, as we show in our paper, HGNNS have distinet parameter sets for each node type (15), edge type (26), and meta-path type & 35). These facts cause HGNNs to learn entirely different feature extractors for nodes and edges of different types in other words, the final embeddings for source and target nodes are computed by different sets of parameters in HGNNs. Thus, a classifier pre-trained on source nodes will fail to perform well on inference tasks for target nodes. The field of domain adaptation (DA) targets this setting, seeking to transfer knowledge from a source domain with abundant labels to target domain which lacks them O 19 20 27). However, distinct feature extractors across node types in HGNNs break standard assumption of DA setting, namely that source and target domains share the same feature extractors (e.g., CNNs for both source and target image domains). As we demonstrate in this paper, in our problem setting, DA approaches fail to achieve the outstanding performance they are known for in computer vision and NLP.
In our work, we first dissect the gradient path of HGNN models to see how feature extractors are designed independently for each node type, and some empirical consequences. Then we theoretically analyze how feature extractors across node types relate to each other and how their output distributions could be represented in terms of each other. We model this theoretical relationship between two feature extractors as Knowledge Transfer Network (KTN) which can be optimized to transform target embeddings to fit the source domain distribution. We perform an extensive evaluation of our method on 18 different transfer learning tasks on HGs where we compare against ttate-of-the-art domain adaptation baselines. Additionally, in order to understand which environments are ideal for transferring knowledge between different node types for HGs, we formulate a synthetic heterogeneous graph generator that allows us to study the sensitivity of these methods.
Our main contributions are:
Novel and practical problem definition: To the best of our knowledge, KTN is the first zero-shot cross-type transfer learning method rumning on a heterogeneous graph - transfer knowledge across different node types within heterogeneous graph.
Generality: KTN is a principled approach analytically induced from the architecture of HGNNs, thus applicable any HGNN models, showing up 960% performance improvement for zero- labeled node inference across 6 different HGNN models.
Effectiveness: We show that KTN outperforms state-of-the-art domain adaptation methods, being up to 73.3% higher in MRR on 18 different transfer learning tasks on HGs.
Sensitivity Analysis: We provide HG generator model analyze how the node attribute and edge distributions of HGs affect the performance of KTN and other methods on the task.
2 Related Work
Various transfer learning problems have been defined on the graph domain. 221 synthetic graphs from unstructured data and transfer knowledge over the graphs using GNNs. On
38 42 construct

---

the other hand, 13: 14: 24: 37) focus on extracting knowledge from the existing graph structures. They pretrain GNN model on source graph and re-use the model on a target graph. While these methods focus on homogeneous graphs, (I6 40) transfer HGNNs across different HGs. However, none of them can be directly applied to our cross-type transfer learning problem running on a single HG. Here we cover two classes of learning approaches that are related to our problem. As HGNNs are the models to which our method can be applied, we cover them extensively in Section 3
Zero-shot domain adaptation (DA) transfers knowledge from source domain with abundant labels to a target domain which lacks them. Zero-shot DA can be categorized into three groups MMDD-based methods, adversarial methods, and pptimal-transport-baseed methods. MMDD-based methods (18) 20, 299 minimize the maximum mean discrepancy (MMD) (II) between the mean embeddings of two distributions in reproducing kernel Hilbert space. Adversarial methods ( 19) are motivated by theory in (2) 3 suggesting that a good cross-domain representation contains no discriminative information about the origin of the input. They learn
features by min-max max game between the domain classifier and the feature extractor. Optimal transport based methods (27) estimate the empirical Wasserstein distance (25) between two domains and minimizes feature extractor network and a task classifier network. Adversarial and OT-based methods use an additional domain classifier network. Due to the assumption that source and target domains have the same modality D the standard DA setting assumes identical feature extractors across domains. More descriptions can be found in Appendix A.9
min-
the distance in an adversarial manner. All three categories rely on two networks
Label propagation (LP) approaches (c.g.. (45) use message- passeepassnngg to pass each node's 's label to its neighbors according normalized edge weights. LP relies only a graph's 's edges, and is therefore easily applied to heterogeneous graph - labels are simply propagated across edges, regardless of type. In this paper we also evaluate similarly-simple baseline, embedding propagation (EP). Similar to LP, EP recursively propagates source embeddings (computed using source labels) until they reach the target domain. EP exploits both node attribute information and the node adjacencies, but only uses the source node embeddings.
3 Preliminaries
In this section we review heterogeneous graphs and heterogeneous graph neural networks (HGNNs).
3.1 Heterogeneous graph
Heterogeneous graphs (HGs) are an important abstraction for modeling the relational data of multi- modal systems. Formally, a heterogeneous graph is defined as G (V, T, R) where the node set V: the edge set consisting of ordered tuples eij := (i,j) with i,j V, where Eij E iff an edge exists from to j; the set of node types T with associated map r V + T; the set of relation types R with associated map Φ : E + R. This flexible formulation allows directed, multi-type edges. We additionally assume the existence of node attribute vector 2; (x))) for each i V. where X, is an attribute matrix specific to nodes of type
3.2 Heterogeneous Graph Neural Networks (HGNN)
43). Fully-specified HGNN models have specialized parameters for each node type 15, edge type (26), and meta-path type (8) to most effectively utilize the complex relationships encoded in the HG data structure. this paper, we use a flavor of HGNN known Heterogeneous Message-Passee Passing Neural Network (HMPNN) our base model on which to demonstrate KTN (though KTN can be implemented in almost any HGNN, as show in experiments in Section 6 The HMPNN merely extends the standard MPNN (T) by specializing all transformation and message matrices in each node/edge type. With its generality, HMPNN is itself base model for RGCN (2) and HGT (15), and is also widely used as default HGNN model in popular GNN libraries (e.g., pyG 7.. TF-GNN (6, DGL (34).
Various HGNN models have been proposed 15 26 35
In our problem, source and target node types could have either (1) different distributions on the same attribute space (2) entirely different attribute spaces
3

---

In HMPNN, for any node j, the embedding of node j at the [-th layer is obtained with the following generic formulation:
h = Transform (2) (Aggregate @ (E()))
(I)
where E()) = ((.
V) denotes all the edges which connect (directionally) to j. The
above operations typically involve type-specific parameters to exploit the inherent multiplicity of modalities in heterogeneous graphs. First, we define a linear Message function:
Message (9 (t.j) = Ma a((ss))) (n--D 14," (1
(2)
where M, are the specific message passing parameters for each edge type E R. and each of L HMPNN layers. Then defining (j) as the set of edges of type r pointing to node j, the Aggregate function mean-pools messages by edge type, and concatenates:
Aggregate ( (E(i)) =
(6,
Σ
Message (e)
(3)
FER
(EE,(5)
Finally, the Transform function maps the message into type-specific latent space:
Transform" G) = a aW 7()
Aggregate (E()))
(4)
where WiOh are the specific transformation parameters for each node type E T and each of HMPNN layers. By stacking I. layers, HMPNN outputs highly contextualized final node represen- tations, and the final node representations can be fed into another model to perform downstream heterogeneous network tasks, such as node classification link prediction.
3.3 Problem definition
Using notations defined above, we formalize our novel transfer learning problem on HGs.
Problem Definition 1. Zero-shot cross- type transfer learning running on a HG:
In given heterogeneous graph G (V. E, T, R) with node attributes X = U,,T assume node types s and t share a classification task {1,, y1) i E Vs, VeJ. During the training phase, using labels {(i, y4) V&} only for source-type nodes, embeddings h; for all nodes E V and a classifier g g(h,) = 0J to predict labels from the node embeddings h4, During the test phase, task is to predict labels { j Vi) of target-type nodes where none of labels of target-type nodes were used for training.
train an HGNN model f fGG
h; get node
4 Cross- Sype Feature Extractor Transformations in HGNNs
We define ft : G + Rd to be the "feature extractor" of a HGNN, which is composed of parameters participating map input node attributes of type I into shared feature space Rd. In this section, we derive a strict transformation between feature extractors within HMPNN. Specifically, for any two nodes i.j with types r(i) and r(i) t. we derive an expression for f. in terms of ft, and use that expression to inspire trainable transfer learning module called KTN in the following section. For simplicity, throughout this section we ignore the activation ax-) within the Transform function
in Equation
4.1 Feature extractors in HMPNNS
We first reason intuitively about the differences between fs and Jr when / 4, using toy hetero- geneous graph shown in Figure I(a) Consider nodes en and v2, noticing that r(1) * 7(2). Using HMPNN's equations
2 from Section 3.2 for any E (0. T. [6-1] 1) we have
h1
(1)
(!)
!
hg
(1-1)
|| M
(1-1) || 14 "]
(5)
h2
= w (1)
[ st (n((-)) || 112 M n) ||
(N((D)
|| 12 ")]
(6)
that 11 and h2 (1) which are features of different types, are extracted using disjoint sets of model parameters at 1-th layer. In a 2-layer HMPNN, this creates unique gradient backpropagation paths between the two node types, as illustrated in Figures ((b) l(c) In other words, even though the same HMPNN is applied to node types s and I, the feature extractors fs and fa have different computational paths. Therefore they project node features into different latent spaces, and have different update equations during training.
where n = Iz. From these equations,

---

type
bype +
V.G.
W
(a) Toy graph
(b) Gradient path for feature extractor f=
(c) Gradient path for feature extractor f+
Figure Illustration of a toy heterogeneous graph and the gradient paths feature extractors fa and fr. Colored arrows in figures (b) and (c) show that the same HGNN nonetheless produces different gradient paths for each feature extractor. Color density of each box in (b) and (c) is proportional to the degree of participation of the corresponding parameter in each feature extractor.
grodients for W
Small gradients
00-
L2 norm of gradients
L2 norm of gradients
No gradients
077
For Ae
No grodients For 1010 AT2
160
100
Number Reraaions
Number of horations
Numberr of Borations
(a) Test accuracy across various feature extractors
(b) L2 norms of gradients W+(+)
(c) L2 norms of gradients of Mo(-)
Figure 2: HGNNs trained on source domain underfit target domain even on a "nice" heterogeneous graph. (a)
Performance the simulated heterogeneous graph for 4 kinds of feature extractors; (source: source extractor f. on source domain, target-sre-path source extractor on target domain, target-org-pont target extractor fe on target domain, and theoretical-Ke KTN: target extractor ft on target domain using KTN.) (b-c) norms of gradients of parameters W and Mat) in HGNN madels.
Empirical gap between fs and ft
Here we study the experimental consequences of the above observation via simulation. We first construct synthetic graph extending the 2-type graph in Figure [[a] have multiple nodes per-type, and multiple classes. To maximize the effects of having different feature extractors, we sample source and target nodes from the same feature distributions and each classes are well-separated in the both the graph and feature space (more details available in Appendix AA..II.
On such a well-aligned heterogeneous graph, without considering the observation in Section 4.1 there may seem be no need for domain adaptation from fe fs. However, when we train the HMPNN model solely on s-type nodes, shown in Figure 2(a) we find the test accuracy for s-type nodes to high (90%, blue line) and the test accuracy for t-type nodes to be quite low (25%, green line). Now if instead we make the t-type nodes use the source feature extractor f.. much more transfer learning is possible (~65%, orange line). This shows that different feature extractors present in the HMPNN model result in the significant performance drop, and simply matching input data distributions can not solve the problem.
To analyze this phenomenon at the level of backpropagation. in Figures 2(b)
2(c) we show the magnitude of gradients passed parameters of source and target node types. As illustrated in
Figures ((b) (cc) we find that the final-layer Transform parameter wx(2) for type-/ nodes have
zero gradients (Figure 2(b),, and similarly for the final-layer Message parameters (Figure 2(c)).
Additionally, those same parameters the first-layer for (-type nodes have much smaller gradients
than their s-type counterparts: (11)) (blue line in Figure 2(b). (!() and ((1) (blue and orange lines in Figure 2(c) appear below than other lines. This is because they contribute to f. less than f.
This case study shows that even when HGNN is trained on relatively simple, balanced, and class-separated heterogeneous graph, model trained only on the source domain node type cannot transfer to the target domain node type.
5

---

We show that a HMPNN model provides different feature extractors for each node type. However, still, fa and ft are built inside one HMPNN model and interchange intermediate feature embeddings with each other. Both H( and H (the output of fs and fi) are computed using the previous layer's intermediate embeddings H H!" and any other connected node type embeddings 4.3 Relationship between feature extractors in HMPNNs
H H²--1) at the L-th HMPNN layer. Therefore H!! and H(!") can be mathematically presented by each other using the (I-- 1) layer embeddings connecting points, so do f. and fe. Based this intuition, we derive a strict transformation between fs and ft, which will motivate the core domain adaptation component of our proposed KTN model.
Theorem 1. Given a heterogeneous graph G =
For any layer > O, define the set of
(I ( 1)-th layer HMPNN parameters as
Q² = 1M : R3 R/U(W!" :tET}.
(7)
Let be the total n X n adjacency matrix. Then for any 8, E T there exist matrices A;, = (A)
and Qis = qto( (Q=(Q(())) such that
A
(8)
where a-o(-) and (-)(-) are matrix functions that depend only s,t.
The full proof of Theorem 1 can be found in Appendix A.1 Notice that in Equation 8 Qi, acts as macro- Transform operator that maps H,()) into the source domain, then A7, aggregates the mapped embeddings into s-type nodes. In other words, Qis acts as mapping matrix from the target domain to the source domain. To examine the implications, we run the same experiment as described in
Section 4.2 while this time mapping the target features H (L) into the source domain by multiplying with Qia in Equation before passing over to task classifier. We see via the red line in Figure 2(a) that, with this mapping. the accuracy in the target domain becomes much closer to the accuracy in the source domain (~70%). Thus, we use this theoretical transformation as a foundation for our trainable HGNN domain adaptation module, introduced in the following section.
4.4 Generalized cross-type transformations for HGNNS
In this section we showed that a HMPNN feature extractor on the (label-abundant) source node type can be expressed in terms of the flabel-scaree target node type feature extractor, and this transformation enables cross-type zero-shot learning for the target node type. As most HGNNs have distinct feature extractors for each node types (even single-layer HGNNs, which have specialized parameters for each node/edge attribute projection layer), they will suffer from the under-trained target embeddings phenomena we showed in Section 4.2 For instance, in the meta- meta-path -pthhaahhhhhh based MAGNN model (8), meta-paths directing toward the target node types are generally less engaged in the source node feature computation and thus receive smaller gradients. While we cannot derive the exact cross-type transformation for all possible HGNNs, the core intuition in the HMPNN theory holds, namely that (2) and H((L) are both computed using the previous layer's intermediate embeddings (see Section 4.3) across all HGNN models. This observation allows to extend our KTN and apply it to almost any HGNN. We illustrate this by applying KTN to different HGNN models in Section where we see greatly increased target-type accuracy.
5 KTN: Trainable Cross-Type Transfer Learning for HGNNs
Inspired by these derivations we introduce our primary contribution, Knowledge Transfer Networks. We begin by noting Equation in Theorem [T a similar form to a single-layer graph convolutional network (17) with a deterministic transformation matrix (Qi.) and a combination of adjacency matrices directing from target node type to source node type (Aī.). Instead of hand-computing the mapping function Qi, for arbitrary HGs and HGNNs (which would be intractable). we learn the
mapping function by modelling Equation g as trainable graph convolutional network, named the

---

Algorithm Training Step on a source domain
Require: heterogencous graph G (V,&, RR), node feature matrices X', source node type target node type adjacency matrix At.. source node label matrix V
Ensure: HGNN I, classifier g, KTN 1KTN
1: H H,( =[( H X)
2: I = [KTN(HH, Ac. IT
LXTN (1:20 -Hi
4: L = (R, ", V.) + ACKTN
5: Update f, using VC
Algorithm 2 Test step target domain
Require: pretrained HGNN f, classifier g. KTN tKIN Ensure: target node label matrix 3
1: H1, [G. HT))
X)
2: Hj = H Th
3: return g(H))
Knowledge Transfer Network, (KTN(-). KTN replaces Qis and Aie in Equation as follows:
(HT) (1) A CKIN ( tKTN (KTN(H! (m)
(10)
where Ars is an adjacency matrix from node type t to and Tis is trainable transformation matrix. By minimizing LKTN, Tis is optimized to a mapping function of the target domain into the source domain.
5.1 Algorithm
We minimize classification loss LaL. and transfer loss LKTN jointly with regard to a HGNN model f, a classifier g. and a knowledge transfer network tRTN as follows:
min Le f. RaNN
|In(G, X), (I))
where À is hyperparameter regulating the effect of LKIN; and f(g, and f(g, denote H!L)) and FT!" respectively. Algorithm describes training step on the source domain. After computing the node embeddings ( and H H!!)). we map H(1h (L) to the source domain using txIN and compute CKTN. Then, update the models using gradients of Lct. (computed using only source labels) and 2 CKTN. Algorithm describes the test phase on the target domain. After we get node embeddings 11(44)
from the trained HGNN model, map ( into the source domain using the trained transformation matrix T... Finally we pass the transformed target embeddings H; into the classifier which was trained on the source domain.
Indirect Connections We note that in practice, the source and target node types can be indirectly connected in HGs via other node types (i.e., there is no At.). Appendix A.2 describes how we can easily extend KTN to cover domain adaption scenarios this case.
Experiments
6.1
Datasets
Open Academic Graph (OAG). dataset introduced in (44) composed of five types of nodes: papers (P), authors (A), institutions (I), venues (V), fields (F) and their corresponding relationships. Paper and author nodes have text-based attributes, while institution, venue, and field nodes have text- and graph structure-based attributes. Paper, author, and venue nodes are labeled with research fields two hierarchical levels, I.1 and 1.2. We construct three field-specific subgraphs from OAG: computer science, computer networks, and machine learning academic graphs
7

---

Table 1: Open Academic Graph Computer Science field. The gain column shows the relative gain of our methad over using no domain adaptation (Base column). o.o.m denotes out-of-memary errors.
Task
Metric
Base
DAN
JAN
DANN
CDAN
CDAN-E
WDGRI.
FP
KTN (gain)
P-A (1.I)
NDCG
0.399
0.452
0.405
0.292
0.262
0.261
0.260
0.178
0.425
0.623 (56%)
MRR
0.257
0.361
0.314
0,179
0.129
0.111
0.138
0.041
0.363
0.629 (112%)
(LI)
MRR
NDCG
0.318
0.508
0.544
0.294
0,229
0220
0.364
0.346
0,020
0.195
0.047
0.153
0.22
0.557
0.507
0.711 (123))
0.733
(L))
NDCG
MRR
0.364
0,459
0.413
0.457
0,470
0.458
0.382
0.341
0.205
0.346
0.253
0.339
0.403
0.327
0.207
0011
0.461
0.389
0.61 (46%)
0.698 (9256)
V-A (LI)
NDCG
0.283
0.483
0.435
0,243
0332
0,438
0272
0.333
0.154
0,584
MRR
0.153
0.365
0.345
0.094
0.241
0.444
0.144
0.066
0.006
0.586 (34025)
P-A (1.2)
NDCG
0.229
0.250
0.339
0.315
0.282 (25/6)
0.121
0.0.m
6.0m
0,0.8
0,020
o.c.m
0.143
A-P
MRR
NDCG
0.095
0.197
0.052
0.162
0.0.m
0.106
0.204
0.032
0.158
0.161
0,045
0.017
0.132
0..mm
0.132
0.208
0.242 (155%)
0.287 (46 (46%))
(L2)
MRR
NDUG
0.310
0.347
0.329
0.296
0.295
0.198
0.223
0.128
02388
0.097
0,273
0.110
0.289
0.0.m
c.c.m
0.227
0,402
(.399 (2976)
(1.2)
NDCG
0.235
0.249
0.251
0.214
0.187
0.944
0.205
0.217
0.085
0.119
0.000
0.166
0.252 (7%L)
Table 2: PubMed graph.
The gain column shows the relative gain over using Base column.
Task
Metriic
Base
DAN
JAN
DANN
CDAN
CDAN-E
WDGRL
L,
EF
KTN (gain)
D-G
MRR
NDCG
0.587
0.425
0.629
0.414
0.615
0.614
0.397
0.624
0.443
0.646
0.604
0.389
0
0.571
0.499 (24%))
0.700 (19%))
G-D
MRR
NDCG
0.596
0.354
0.599
0.362
0.332
0.577
0.356
0.599
0.337
0.581
0.606
0.362
0.340
0.578
0.351
0.576
0.353
0.580
0.662 (11%)
0.445 (26%))
PubMed. A network composed of of four types of nodes: genes (G), diseases (D), chemicals
(C), and species (S), and their corresponding relationships. Gene and chemical nodes have graph structure-based attributes, while disease and species nodes have text-based attributes. Each gene and disease labeled with a set of diseases they belong to or cause.
Synthetic heterogeneous graphs. We generate stochastic block models ① with multiple node/edge types. We label each node types with the same set of classes. Then we control feature/edge distribu- tions within/between node types by manipulating feature/edge signal-to-noise ratio within/between classes. complete definition of the generative model is given in Appendix A.7
6.2 Baselines
We compare KTN with two MMD-based DA methods (DAN (8)) JAN (20)), three adversarial DA methods (DANN @). CDAN (1)) CDAN-E (I)), one optimal transport- based method (WD- GRL (27)). and two traditional graph mining methods (LP and EP (5)). For KTN and DA methods. we use HMPNN as their base HGNN model. More information of each method in Appendix A.9
6.3 Zero-shot transfer learning
We run 18 different zero-shot transfer learning tasks across three OAG and PubMed graphs. We run each experiment 3 times and report the average value. Due to the space limitation, we report the standard deviations and results on OAG OAG-computer networks and OAG-machin learning in Appendix 3..3 Each heterogeneous graph has the same node classification task for both source and target node types. During training, we are given 1) the heterogeneous graph structure information (i.e., adjacency matrices), 2) input node attribute matrices for node types, and 3) labels on source-type nodes for the classification task. During the test phase, we predict labels on target-type nodes for the same classification task. The performance is evaluated by NDCG and MRR - widely adopted ranking metrics [14 15).
In Tables I and 2 our proposed method KTN consistently outperforms all baselines on all tasks and graphs by up to 73.3% higher in MRR (P-A(L1) task in OAG-CS, Table J.. When we compare with the base accuracy using the model pretrained on the source domain without any domain adaptation (3rd column, Base), the results are even more impressive. We see our method KTN provides relative gains of up to 340% higher MRR without using any labels from the target domain. These results show the clear effectiveness of KTN zero-shot transfer learning tasks on heterogeneous graph. We mention that venue and author node types are not directly connected the OAG graphs (Figure (5h) in Appendix), but KTN successfully transfer knowledge by passing intermediate node types.
8

---

Table 3: KTN on different HGNN models. The Source column shows accuracy for source node types. Base and KTN columns show accuracy for target node types without/with using KTN, respectively. The Gain column shows the relative gain of our method over using no domain adaptation.
P-A (LD
A-P (LI)
HGNN
Metric
Source
Base
KTN
Gain
Source
Base
KTN
Gain
R-GUN
XDCG
0,765
0.337
0.377
71.12%
0.618
0.365
0.647
66.2%
148.73%
0.111
HAN
NDCG
0,476
0.179
0.520
190.56%
0.515
0.182
0.512
181.33%
MKR
0,416
0.047
0.497
900.55%
0.509
0:055
0,527
850.90%
0.581
10133%
MRR
0.749
0.178
0.563
216.17%
0.670
0.149
0.565
279.525
MAGNN
NDOG
0.657
0.463
0.574
24.01%
0.676
0.557
0.622
11.6%
0.592
MPAN
NDCG
0.002
0.445
0.590
33.11%
0.646
0.307
0.021
101.92%
MRR
0.572
0.319
0,375
80.10%
0.660
0.145
0.595
511.42%
HMPNN
NDOG
82.88 F2
MRR
0.777
0.297
0.629
111.86%
0.661
0.318
0.711
123,30%
Among baselines, MMD-based models (DAN and JAN) outperform adversarial- based methods (DANN, CDAN, and CDAN-E) and optimal transport- based method (WDGRL.), unlike results reported in 19 27). These reversed results are a consequence of HGNN's unique feature extractors for each domains. Adversarial- and optimal transport-based methods define separate losses for source and target feature extractors (which are not separated in their shared feature extractor assumption), resulting in divergent gradients between different feature extractors and poor domain adaption performance. This shows again the importance of considering different feature extractors in HGNNs. More analysis can be found in Appendix A.4
Baseline Performance.
6.4 Generality KTN
Here, we use 6 different HGNN models, R-GCN (26), HAN (35) HGT 15 MAGNN (8. MPNN (I0) and HMPNN. MPNN maps all node types to the shared embedding space using projection matrices at the beginning then applies MPNN layers designed for homogeneous graphs. In Table 3 KTN improves accuracy on the target nodes across all HGNN models by up to 960%. This shows the strong generality of KTN. More results and analysis can be found in Appendix A.5
6.5 Sensitivity analysis
Using our synthetic heterogeneous graph generator, we generate non- non-trivial 2-type heterogeneous graphs to examine how the feature and edge distributions affect the performance of KTN and other baselines. We generate a range of test-case scenarios by manipulating (1) signal-to-noise ratio a, of within-class edge distributions and (2) signal-to-noise ratio oy of within-class feature distributions across all of the (a) source-source (s ++ s), (b) target-target (1 <> l), and (c) source-target (s <> I) relationships.
For instance, in Figure 3 for each edge type (s + s, t t, and +> t, differentiated by colors), there are two different types of edges, edges within the same class (plain line) and edges across different classes (dotted line). For each edge type, we manipulate o. by chang- ing the ratio of within-class and cross-class edges, and of by diverging feature distribu- tions between classes. Thus there will be signal-to-noise ratios in total. higher signal- to-noise ratio for a particular data dimension (edges or features) across particular relation- ship E {s + 8, f ++ t. ++ t means that classes are more separable in that data dimension, when comparing within r, and hence easier for HGNNs. Note that while tuning one (.) on the range [1.0, 10.0], the remaining five o()) are held at 10.0. Additionally, we vary (()) across two scenarios: (I) "easy": source and target node types have same number of clusters and same feature dimensions, (II) "hard" source and target node types have different number of clusters and feature dimensions. Note that clusters and classes are different concepts in this experiment; several clusters could have the same class label.
Blw same/ same/ differenn node types
Biw same/ same/differen differeen classes (e.g.. A, B, C)
Source node type
Target node type
's'
'P'
Figure 3: Synthetic HG generator

---

KTN
13
(a) Edge dist. (easy)
(b) Feature dist. (easy)
(c) Edge dist. (hard)
Feature dist. (hard)
Figure 4: Effects of edge and feature distributions across classes and types in heterogeneous graphs.
Figures 4(a) and [(c) [[c) results from changing o across the three relation types. We see that KTN affected only by de across the ++ (cross-types) relationship, which accords with our theory. since KTN exploits the between-type adjacency matrix. Surprisingly, as seen in Figures 4(b) and 4(d)
do not find a similar dependence of KTN on of, which shows that KTN robust by learning purely from edge homophily in the absence of feature homophily. Regarding the performance of
other baselines, EP shows similar tendencies as KTN- only affected by cross-type de because EP also relies on cross-type propagation along edges. However, its accuracy is bounded above due to fact that does not exploit the (unlabelled) target features. DAN and DANN, which do not exploit cross-type edges, are not affected by cross-type de. However, they show either low or unstable performance across different scenarios. DAN shows especially poor performance in the "hard" scenarios (Figure FFc) and 4(d), failing to deal with different feature spaces for source and target domains.
7 Conclusion
In this work, proposed the first cross-type zero-shot transfer learning method for heterogeneous graphs. Our method, Knowledge Transfer Networks (KTN) for Heterogeneous Graph Neural Networks, transfers knowledge from label-abundant node types to label-scarce node types. We illustrate KTN handily improves HGNN performances up to 960% for zero-labeled node types across 6 different HGNN models and outperforms many challenging baselines up 73% higher in MRR. Future work in the area includes filtering noisy edges between source and target domains and making KTN more robust and less dependent structure of given noisy heterogeneous graphs.
Limitation Statement Our transfer learning method is limited to node types sharing the same task (i.e., the same classifier). We plan to extend work to transfer knowledge between different tasks running on different node types on heterogeneous graph.
Societal Impact Statement KTN allows organizations to learn better from their own graph data, leveraging its structure without requiring external information. We believe this has a number of positive applications (preserving model quality without needing extra datasets). However like all modeling improvements, its true impact depends on what modeling tasks the technique applied to.
8
Acknowledgement
MY gratefully acknowledges support from Amazon Graduate Research Fellowship. GPUs partially supported by AWS Cloud Credit for Research program.
References
[1] Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research, 18(1):6446-655 6531, 2017.
[2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. theory learning from different domains. Machine learning, 19(1):151-175, 2010.
10

---

[3] Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of represen- tations for domain adaptation. Advances in neural information processing systems, 19:137, 2007.
[4] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[5] Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 135-144, 2017.
[6] Oleksandr Ferludin, Amo Eigenwillig, Martin Blais, Dustin Zelle, Jan Pfeifer, Alvaro Sanchez- Gonzalez, Sibon Li, Sami Abu-El-Haija, Peter Battaglia, Neslihan Bulut, et al. Tf-gnn: Graph neural networks in tensorflow. arXiv preprint 22Xiv:220..03522, 2022.
[7] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint 1eXXvv:903.0222 2019.
[8] Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. Proceedings of The Web Conference 2020, pages
2020.
[9] Yaroslav Ganin, Evgeniya Ustinova, Hama Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarte training of neural networks. The journal of machine learning research, 17(1):2096-200 2030, 2016.
[10] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning. pages 1263-1272. PMLR, 2017.
[11] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773 2012.
[12] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning large graphs. In Proceedings of the st International Conference on Neural Information Processing Systems, pages 1035, 2017.
[13] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre- pre-training graph neural networks, arXiv preprint arXiv: 1et.. 2019.
[14] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867, 2020.
[15] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings The Web Conference 2020, pages 2704-2710, 2020.
[16] Tiancheng Huang, Ke Xu, and Donglin Wang. Da-hgt: Domain adaptive heterogeneous graph transformer. arXiv preprint arXiv:2012.05668) 2020.
[17] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint a (et::::000002,00 2016.
[18] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97- PMLR, 2015.
[19] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. arXiv preprint 171vv1700..16667, 2017.

---

[20] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pages 2208-2217 PMLR, 2017.
[21] Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh. Progressive graph learning for open-set domain adaptation. In International Conference on Machine Learning, pages 6468 6478. PMLR, 2020.
[22] Xinhong Ma, Tianzhu Zhang, and Changsheng Xu. Gcan: Graph convolutional adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 0266-82766 2019.
[23] Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean, Distributed repre- sentations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111 3111-3119, 2013.
[24] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang. Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1150-1160, 2020.
[25] levgen Redko, Amaury Habrard, and Marc Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 737-753. Springer, 2017.
[26] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer, 2018.
[27] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Thirly-Second AAAI Conference on Artificial Intelligence, 2018.
[28] Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang, An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pages 243-246, 2015.
[29] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[30] Yizhou Sun and Jiawei Han. Mining heterogeneous information networks: principles and methodologies. Synthesis Lectures on Data Mining and Knowledge Discovery, 3(2):1-159, 2012.
[31] Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnetminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 990-998, 2008
[32] Anton Tsitsulin, John Palowitch, Bryan Perozzi, and Emmanuel Müller. Graph clustering with graph neural networks. arXiv preprint arXiv:2006. 16904, 2020.
[33] Anton Tsitsulin, Benedek Rozemberczki, John Palowitch, and Bryan Perozzi. Synthetic graph generation to benchmark graph learning. WWW'2/, Workshop on Graph Learning Benchmarks, 2021.
[34] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et al. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.03315 2019.
[35] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Het- erogeneous graph attention network. In The World Wide Web Conference, pages 2022-2032, 2019.
12

---

[36] Thomas Wolf. Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue. Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Langugge Processing: System Demonstrations, pages 38 45, 2020.
[37] Man Wu, Shirui Pan, Chuan Zhou, Xiaojun Chang, and Xingquan Zhu. Unsupervised domain adaptive graph convolutional networks. In Proceedings of The Web Conference 2020, pages 1457-1467, 2020.
[38] Qitian Wu, Chenxiao Yang, and Junchi Yan. Towards open-world feature extrapolation: An inductive graph learning approach. Advances in Neural Information Processing Systems, 34:19435-19447, 2021.
[39] Carl Yang, Yuxin Xiao, Yu Zhang. Yizhou Sun, and Jiawei Han. Heterogeneous network representation learning: A unified framework with survey and benchmark. IEEE Transactions on Knowledge and Data Engineering. 2020.
[40] Shuwen Yang, Guojie Song, Yilun Jin, and Lun Du. Domain adaptive classification hetero- geneous information networks. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 1410-1416, 2021.
[41] Yaming Yang, Ziyu Guan, Jianxin Li, Wei Zhao, Jiangtao Cui, and Quan Wang. Interpretable and efficient heterogeneous graph convolutional network. IEEE Transactions on Knowledge and Data Engineering, 2021.
[42] Jiaxuan You, Xiaobai Ma. Yi Ding, Mykel Kochenderfer, and Jure Leskovee. Handling missing data with graph representation learning. Advances in Neural Information Processing Systems, 33:19075-19087, 2020.
[43] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh Chawla. Het- erogeneous graph neural network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 793 803, 2019.
[44] Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang. Bin Shao, Rui Li, et al. Oag: Toward linking large-scale heterogeneous entity graphs. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2585-2595, 2019.
[45] Xiaojin Zhu. Semi-supervised learning with graphs. Carnegie Mellon University, 2005.
13


Reviews:
[{"review_id": "uY2uHcJcWz", "reviewer": "Reviewer_deGV", "summary": "This paper studies the zero-shot node types in knowledge transfer and proposes a model to transfer knowledge from label-abundant node types to zero-labeled ones. The main contributions of this paper lie in the proposed practical scenario and problem definition, i.e., knowledge transfer of zero-shot node types. The transferability of node label knowledge is theoretically analyzed. A node type label knowledge transfer network is proposed.", "questions": "This work proposes a new practical problem definition, which is beneficial to the development of the field. But I have a few confusing questions about the paper. Please point out if I misunderstood.\n\n- In the problem definition, the authors say that only source type labels are available. Although the target labels are not used as training samples, the training graph data seems to contain the labels of the target type (i. target type related aggregation and ii. output embeddings of the nodes of the target type). Is this reasonable? This is my biggest concern.\n\n- Does the proposed method still work on other graph tasks (such as link prediction) and on more complex graphs (such as knowledge graphs)? No need to add experiments, just give a brief statement about feasibility.\n\n- If needed, you can respond to the above *Weaknesses* to help me make a better assessment.", "limitations": "N/A.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper studies a novel and practical problem, and the motivation is clearly stated.\n\n- The proposed graph neural network is supported by theoretical analysis, although some of the analysis is based on a simple toy graph.\n\n- Experimental results show that the proposed method outperforms the baseline methods.\n\nWeaknesses:\n\n- The title is misleading. There is a lot of transferable knowledge in the graph data, but this paper only discusses a specific setting of the node type labels.\n\n- The baseline methods are from 2005–2018. Some new work needs to be discussed.\n\n- As a work that proposes a new problem setting, Section 6.3 needs to provide more details of the experimental setting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Y4XqWMjv1RC", "reviewer": "Reviewer_sHGC", "summary": "This paper proposes a novel domain adaptation method on heterogeneous graphs by reconstructing the source node embeddings through target node embeddings. Theoretical proofs verify the correctness of proposed method. Experiments show that the proposed method achieves significant performance promotion compared with bare HGNN model. ", "questions": "1. Authors should more clearly state how the gradients are computed in Figure 2 (b)-(c), i.e. they are gradients under loss $\\mathcal{L}_s$.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n\n- This paper is good structured and quite informative. Nice writing.\n- Figure 1 gives a simple and clear example, demonstrating how the HGNN is trained with respect to different node types. The combination of Figure 1 and Figure 2 justifies the motivation of this paper.\n- Theoretical results in Section 4.3 is intuitive, easy to understand. Theorem 1 induces an elegant algorithm in Section 5.\n- Extensive experiments validate the effectiveness (Table 1 & 2) and generality (Table 3) of the proposed method.Weaknesses: \n    \nWeaknesses：\n- The methods compared in Table 1 seem to be out-dated. Some recents works could be considered, i.e. Adversarial-Learned Loss for Domain Adaptation (AAAI2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Cc60WvvRxI", "reviewer": "Reviewer_Rrs6", "summary": "This paper notes that many HG datasets suffer from label imbalance between node types, and proposes a zero-shot transefer learning module to transfer knowledge from label-abundant node types to zero-labeled node types. This is a good work in my opinion.", "questions": "Please respond to the weaknesses raised above.", "limitations": "The authors have well discussed the limitations and potential negative social impact of this work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n  - This work notes a good research problem. The issue of label imbalance among different node types is common in existing HG datasets.\n  - The proposed method KTN has a solid theoretical foundation, and its implementation is quite easy.\n  - The experimental results show that KTN is very effective in improving the performance of existing HGNN models.\n  - The provided source codes and datasets in the supplemental material facilitates the good reproducibility of this work.\n\n- Weaknesses\n  - In section 3.2, the authors have reviewed some HGNN methods, but one recent HGNN method is missing, i.e., *[TKDE 2021] Interpretable and Efficient Heterogeneous Graph Convolutional Network*.\n  - For the training step and the test step, why the formula in Line 2 of Algorithm 1 has the adjacency matrix $A_{ts}$ while the formula in Line 2 of Algorithm 2 does not?\n  - Is HMPNN an existing work? If so, it should be cited. The authors state that HMPNN merely extends the standard MPNN, but why does the message function (Eq. 2) in this work look different from the message function (Eq. 1) in MPNN? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "1HbmxXCFuyE", "reviewer": "Reviewer_AyUC", "summary": "This paper aims to transfer knowledge from the source node type with abundant labels to the target node type with zero labels on heterogeneous graphs by proposing a knowledge transfer network. In particular,\n* The authors introduce the undiscovered problem of transferring knowledge from label-abundant nodes to zero-labeled nodes, which are connected via edges over the heterogeneous graph.\n* The authors tackle this problem with a novel knowledge transfer network, which trains a feature transformation matrix that maps representations for target nodes with zero-label to the source node representation space.\n* The authors propose a knowledge transfer network with a motivating theoretical observation that there exist transformation matrices to exactly map the target node representations to the source node embedding space, for predicting target zero-labeled nodes over the learned source-node representation space.\n* The experimental results show that the proposed knowledge transfer network significantly outperforms the no domain adaptation and existing domain adaptation methods, by a large margin.", "questions": "### Major Questions and Suggestions\n* The authors may discuss the relevant work on transfer learning or domain adaptation on graphs with graph neural networks (See the first weakness above).\n* The authors should improve the explanations on synthetic graph experiments (See the second weakness above).\n* The authors do not explain the task (e.g., P-A (L1)) in Table 1 and Table 2. I guess P-A denotes the Paper (Source) and Author (Target) as node types, however, the authors should define this.\n* It would be interesting to see the performance on few-shot tasks, where few target nodes have their labels.\n\n### Minor Questions and Suggestions\n* The description of the first contribution in Line 76 is a little bit unclear. It is better to specify the term \"cross-type transfer learning\".\n* Does the number of batchs in Figure 2 denote the number of iterations, or the batch size? It seems it would be the number of iterations, but the current term of the number of batchs is ambiguous.\n* I am just curious that, instead of minimizing the loss for KTN for the representation of the last layer (i.e., L), why the authors do not minimize the KTN losses for all layers (i.e., 1, 2, 3, ..., L). \n* A caption for Table 2 is not informative.\n\n### Typo\n* Line 88: in in Section 3.", "limitations": "The authors provide the limitations and potential societal impact of their work. While the authors do not describe the *negative* societal impact, but only describe the positive point, this does not hurt the contributions of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* This paper tackles the highly realistic problem of zero-shot knowledge transfer from source to target nodes on a heterogeneous graph, which has, to my knowledge, not been introduced so far.\n* The proposed method of representing zero-labeled target nodes in terms of label-abundant source nodes with the transformation matrices from target to source is convincing and highly interesting. I enjoy reading this method.\n* The proposed architecture, namely knowledge transfer networks, is well-motivated by empirical observations to a toy heterogeneous graph, and by theoretical derivations to feature extraction modules on heterogeneous graph neural networks. \n* The performance improvements from baselines are significant and impressive.\n* This paper is extremely well written and easy to follow. \n\n### Weaknesses\nNote that the below points are not the major limitations of this work, but I would appreciate if the authors address them.\n* The authors may include the work [1, 2] that considers transfer learning (or domain adaptation) problems for graph-structured data with graph neural networks, and perhaps more work on this topic of graph transfer learning could be discussed.\n* The explanations of synthetic graph experiments in Section 4.2 and Section 6 are unclear. Also, the explanation in Appendix A.7, which contains full descriptions, is confusing as well. This might be because the setting for heterogeneous graphs differs from normal graphs, and I think it would be better to illustrate the authors' synthetic graph experimental setup with Figure.\n\n---\n\n[1] You et al. Handling Missing Data with Graph Representation Learning. NeurIPS 2020.\n\n[2] Wu et al. Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach. NeurIPS 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept