Descent Steps of a Relation-Aware Energy Produce Heterogeneous Graph Neural Networks
Authors: Hongjoon Ahn
Yongyi Yang
Quan Gan
Taesup Moon
David Wipf
Venue: NeurIPS 2022 Accept (2022)
Keywords: heterogeneous graph neural networks
bilevel optimization
unfolded optimization

Abstract: Heterogeneous graph neural networks (GNNs) achieve strong performance on node classification tasks in a semi-supervised learning setting. However, as in the simpler homogeneous GNN case, message-passing-based heterogeneous GNNs may struggle to balance between resisting the oversmoothing that may occur in deep models, and capturing long-range dependencies of graph structured data. Moreover, the complexity of this trade-off is compounded in the heterogeneous graph case due to the disparate heterophily relationships between nodes of different types. To address these issues, we propose a novel heterogeneous GNN architecture in which layers are derived from optimization steps that descend a novel relation-aware energy function. The corresponding minimizer is fully differentiable with respect to the energy function parameters, such that bilevel optimization can be applied to effectively learn a functional form whose minimum provides optimal node representations for subsequent classification tasks.  In particular, this methodology allows us to model diverse heterophily relationships between different node types while avoiding oversmoothing effects.  Experimental results on 8 heterogeneous graph benchmarks demonstrates that our proposed method can achieve competitive node classification accuracy.

Descent Steps of a Relation-Aware Energy Produce Heterogeneous Graph Neural Networks
Hongjoon Ahn'; Yongyi Yang"; Quan Gan 3, Taesup Moon If and David Wipf 3t ECE/IPA//ASRU/NNCC, Scoul National University, University of Michigan, Amazonn Web Services ( (hong0805, smonn @nuu. ac. kz, yongyiiggeeeoo @umicuhhh chncchhco edu, quagant amazon. com, davidwipf
com
Abstract
Heterogeneous graph neural networks (GNNs) achieve strong performance on node classification tasks in a semi-supervised learning setting. However, in the simpler homogeneous GNN case, message- message-passing passing- heterogeneous GNNs may struggle to balance between resisting the oversmoothing that may occur in deep models, and capturing long-range dependencies of graph structured data. Moreover, the complexity of this trade-off is compounded in the heterogeneous graph case due to the disparate heterophily relationships between nodes of different types. To address these issues, we propose novel heterogeneous GNN architecture in which layers are derived from optimization steps that descend a novel relation-aware energy function. The corresponding minimizer is fully differentiable with respect to the energy function parameters, such that bilevel optimization can be applied to effectively learn a functional form whose minimum provides optimal node representations for subsequent classification tasks. In particular, this methodology allows us to model diverse heterophily relationships between different node types while avoiding oversmoothing effects. Experimental results 8 heterogeneous graph benchmarks demonstrate that our proposed method can achieve competitive node classification accuracy. The source code of our algorithm is available https://ggthuu..commoooooon/////
1 Introduction
Graph structured data, which contains information related entities as well the interactions between them, arise in various applications such as social networks or movie recommendations. In a real world scenario, these entities and interactions are often multi-typed, such as actor, director, keyword, and movie in movie review graph (20] or authors, papers, terms, and venue in an academic network [31]. Hence the relationships between entities can potentially be much more complex than within traditional homogeneous graph scenario, in which only single node and edge type exists. To this end, heterogeneows graphs have been proposed as a practical and effective tool to systematically deal with such multi-typed graph-structured data. With the advent of graph neural networks (GNN) for instantiating deep models that are sensitive to complex relationships between data instances, is natural that numerous variants have been developed explicitly handle heterogeneous graphs 2 [5 7 10, g 33 37], with promising performance on downstream tasks such as node classification link prediction.
In this vein, message-passinn GNNs represent one of the most widely-adopted heterogeneous architectures, whereby sequence of type-specific graph propagation operators allow neighboring nodes to share and aggregate heterogeneous information 7 29 33). However, in some cases where
Work completed during internship the AWS Shanghai Al Lab. Co-corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

oong-range dependencies exist between nodes across the graph, stacking too many layers to reflect a larger neighborhood may contribute to a well-known oversmoothing problem, in which node features converge to similar, non-discriminative embeddings [2]. In the case of homogeneous graphs, this oversmoothing problem has been addressed (among other ways) using modified GNN architectures with layers patterned after the unfolded descent steps of some graph-regulariedd energy 88 36]. Provided minimizers of this energy remain descriminative for the task at hand, arbitrarily deep networks can be trained without risk of oversmoothing, with skip connections between layers organically emerging to favor node-specific embeddings. However, extending this approach to handle heterogeneous graphs is not straightforward, largely because node/edge heterogeneity (e.g., node- specific numbers of classes or feature dimensions, multiple types of relationships between nodes) is not easily reflected in the vanilla energy functions adopted thus far for the homogeneous case. In particular, existing energy functions are mostly predicated on the assumption of graph homophily, whereby neighboring nodes have similar labels. But, in the heterogeneous case, this assumption is no longer realistic because the labels between nodes of different types may exhibit complex relationships.
To this end, we first introduce novel heterogeneous GNN architecture with layers produced by the minimization of a relation-aware energy function. This energy produces regularized heterogencous node embeddings, relying relation-dependent compatibility matrices as have been previously adopted in the context of belief propagation on heterogeneous graphs, e.g., ZooBP [6]. Secondly, derive explicit propagation rules for our model layers by computing gradient descent steps along the proposed energy, with quantifiable convergence conditions related to the learning rate being considered as well. Furthermore, we provide interpretations of cach component of the resulting propagation rule, which consists of skip-connections, linearly transformed feature aggregation, and self-loop transformation. With respect to experiments, our contributions show that our algorithm outperforms state-of-the-ar baselines on various benchmark datasets. And secondly, we analyze the the effectiveness of using the compatibility matrix and the behavior of the unfolding step as the number of propagations increases.
twofold. First,
2 Existing Homogeneous GNNs from Unfolded Optimization
Consider a homogeneous graph G (V.E), E). with n |V] nodes and |8| edges. We define E Rux= as the Laplacian of G, meaning L = D A, in which D and A are degree and adjacency matrices, respectively. Unfolded GNNs incorporate graph structure by attaining optimized embeddings Y E Randd that are functions of some adjustable weights W. i..., Y = Y (YW B where (W))//W is computable by design. These embeddings are then inserted within an opplication-specife loss
6 (0, W
Σ D(g V (W): tt),
in which g : Rd + R is some differentiable node-wise function with parameters O and c-dimensional output tasked with predicting ground-truth truth node-wise targets t, E R Additionally, y (W) is the i-th row of Y (W). n'' < n is the number of labeled nodes (we assume w.L.og.. that the first n' nodes are labeled), and D is discriminator function, e.g., cross-entropy for node classification, as will be our focus. Given that (W) is differentiable by construction, we can optimize Co (0, W) via gradient descent (over both 0 and W if desired) to obtain our final predictive model.
At a conceptual level, what differentiates GNNs inspired by unfolded optimization is how (*) is obtained. Specifically, these node embeddings are chosen to be the minimum of lower-level, 38 40) (a similar idea has also recently been applied Orignally inspired by [39], the most
graph-regularized energy function [3] 19 23 36
to complete graphs within Tranformer architectures common selection for this energy is given by
[35]
&x (Y) e Y fX:: W) + Atr [Y
(2)
where À > 0 is a trade-off parameter, Y E Ruxd oi is a matrix of trainable embeddings, meaning d-dimensional features across n nodes, and I (X; W) is a base model parameterized by W that produces initial target embeddings using the do -dmeeniiona node features X E x Minimization of (2) over Y results in new node embeddings Y characterized by a balance between similarity
For brevity, will frequently omit this dependency on W when the context clear.

---

with (X; W) and smoothness across graph edges as favored by the trace term tr [YTLY) Euster lly. 2 where ys is i-th row of Y. While (2) can be solved in a closed-form as
Y*()) = arg min (Y = +I) SXX;
for large graphs this analytical solution is not practically computable. Instead, we may form initial node embeddings Y (X: W), and then apply gradient descent converge towards the minimum, with the k-th iteration given by
Y = ((- = a [AL - w):
(4)
which 9 is the step size. Given that is generally sparse, computation of (4) can leverage efficient sparse matrix multiplications. Combined with the fact that the loss is strongly convex, and provided that a is chosen suitably small, we are then guaranteed to converge towards the unique global optima with efficient gradient steps, meaning that for some K sufficiently large Y(K R (W)
Critically, Y(K) (W) remains differentiable with respect to W, such that we may substitute this value into TT in place of Y (W) to obtain an entire pipeline that is end-to-end differentiable w.r.t. W. Moreover, per the analyses from [1 23
40). the k-th unfolded iteration of (4) can be interpreted as an efficient form of GNN layer. In fact, for certain choices of the step size, and with the incorporation of gradient pre-conditioning and other reparameterizattoon factors, these iterations can be exactly reduced to popular canonical GNN layers such as those used by GCN [4. APPNP [8). and others. However, under broader settings, we obtain unique GNN layers that are naturally immune to oversmoothing, while nonetheless facilitating long-range signal propagation across the graph by virtue of the anchoring effect of the underlying energy function. In other words, regardless of how large K is, these iterations/layers will converge to node embeddings that adhere to the design criteria of (2) This is the distinct appeal of GNN layers motivated by unfolded optimization, least thus far in the case of homogeneous graphs.
36
38
3 New Heterogeneous GNN Layers via Unfolding
In this section, we explore novel extensions of unfolded optimization to handle heterogeneous graphs, which, as mentioned previously, are commonly encountered in numerous practical application domains. After introducing our adopted notation, we first describe two relatively simple, intuitive attempts accommodate heterogeneous node and edge types. We then point out the shortcomings of these models, which are subsequently addressed by our main proposal: a general-purposs energy function and attendant proximal gradient steps that both (i) descend the aforementioned energy, and (ii) in doing so facilitate flexible node- and edge-type dependent message passing with non-linear activations where needed. The result is a complete, interpretable heterogencous GNN (HGNN) architecture with layers in one-to-one correspondence with the minimization of heterogencous graph-regularized energy functionn
Before we begin, we introduce some additional notation. An undirected heterogeneous graph G (V,E) is a collection of node types S and edge types T such that V - V/, and È = UleT & Here V, denotes Furthermore, we use Tas' to refer to the set of edge types connecting node types and s'. Note that T includes both the canonical direction, & Tas, and the inverse direction, tin E Te's, that corresponds with type é edge
of ni Vall nodes of type 8, while & represents a set of edge type I.
3.1 First Attempts at Heterogeneous Unfolding natural starting point for extending existing homogeneous models to the heterogeneous case would be to adopt the modified energy
(Y
Σ SES
Y. - IX,, Wo)l
Σ 1 ir (Y F'SIET
(5)
in which L2 is the graph Laplacian involving all edges of relation type and W, parameterizes a type-specific transformation of node representations. Here, we have simply introduced type-specific
For example, if E Text is "author"", then tim Tata is ("author"- "aaaper""""aaper")
3

---

transformations of the initial node representations, as well as adding separate graph regularization term for each relation type t. However, this formulation is actually quite limited can be reduced to the equivalent energy
(x))= = || x fl + +2)((TCC)) with C- C-ΣE L,, SESS'ESIET..
(6)
in which X represents the original node features concatenated with the node type, and is some function of the augmented features parameterized by W - [Wa)ses. Note that by construction, C will necessarily be positive semi-definite, and hence we may conclude through (6) that (5) defaults to the form of a standard seudratically-reguaaizzz
loss, analogous to (2), that has often been applied to
graph signal processing problems [12). Therefore the gradient descent iterations of this objective will closely mirror the existing homogeneous unfolded GNN architectures described in Section 2 to capture the nuances of heterogeneous data.
and fail
To break the symmetry that collapses all olaaiin-sspecifie regularization factors into tr (YTCY), natural option is to introduce t-dependent weights as in
(Y) A
Σ SES
2 Yr f(x:: W. 1|
Σ Σ ir (Y ( LYTM) ''ES EET
in which the set {M,} is trainable. While certainly more expressive than (5) this revision is nonetheless saddled with several key limitations: (i) Unless additional constraints are included on (M,) (e.g.. PSD, etc.), (7) may be unbounded from below; (ii) While M, may be asymmetric, w.l.o.g. the penalty can be equivalently expressed with symmetric weights, and hence the true degrees of freedom are limited This model could be prone overfitting, since during training it could be that M. + O in which case the graph-based regularization is turned off and (iv) Because tr [Y L, YM,] = Ea.)EE (Usi yer)) M, (Ysi Uss)), the energy function (7) is symmetric w.r.t. the order of the nodes in the regularization term, and therefore any derived message passing must also be symmetric, ie., we cannot exploit asymmetric penalization aligned with heterogeneous relationships in the data.
Given then that both (5) and 7 have notable shortcomings, it behooves us to consider revised criteria for selecting a suitable heterogeneous energy function. We explore such issues next.
3.2 A More Expressive Alternative Energy Function
As an entry point for developing a more flexible class of energy functions that is sensitive to nuanced relationships between different node types, we consider previous work developing various flavors of both label and belief propagation 66 27
34 39]. In the more straightforward setting of simple iterative
homogeneous graphs under homophily conditions, label propagation serves process for smoothing known training labels across edges to unlabeled nodes in such a way that, for a given node i, the predicted node label & will approximately match the labels of neighboring nodes Ni. i.e. & Rv i, when j E NG. An analogous relationship holds when labels are replaced with beliefs (27
However, in broader regimes with varying degrees of heterophily, it is longer reasonable for such simple relationship to hold, as neighboring nodes may be more inclined to have dissimilar labels. To address this mismatch, a compatibility matrix H must be introduced such that now label propagation serves to instantiate L, R i for j E Nr (34 (here we treat each i, as row vector). If each predicted label ł, is approximately a one-hot vector associated with class membership probabilities. then the (k, 1)-th element of H roughly determines how a node of class & influences neighbors of class And if we further extend to heterogeneous graphs as has been done with various forms of belief propagation (6, it is natural to maintain a unique (possibly non-square) compatibility matrix
'Note that M.- 1/2 (M, + M, + (M,- - M.") for any weight matrix M, Since y (M, M, yy 0
for any and M,, the penalty actually only depends on the symmetric part (M; + M, ).
'For example, suppose the node features of training nodes are highly correlated with the labels, or actually are the labels (as in some prior label propagation work). Then the model could just learn M, = 0 and achieve perfect reconstruction on the training nodes to produce zero energy, and yet overfit since graph propagation is effectively turned off.

---

H2 for each relation type t, with the goal of finding predicted labels (or beliefs) of each node type satisfying tu s H for s, E S, t E Tas, and (i. E &g. From a practical standpoint, each such H, can be estimated from the data using the statistics the labels (or beliefs) of nodes sharing edges of type t.
Returning to our original goal, regularization factor for learning heterogenerous graph node representations (as required by HGNNs). Specifically, given s' E S. L. E Tax' and (i, &t, we seek enforce y.,H, RS ya's which then naturally motivates the energy
similar intuitions to guide the design of an appropriate
Y. - -(x,; W. W(,
A Σ J'ESIET,,' (A.IEE,
ΣE
Iy y-sll (8)
2
#ES
where À denotes a trade-off parameter, X, E R. represents dos dimeenssiia initial features nodes for node type s. Note
and Y, E IR,, is the embedding of di-ddimesio features that Ysi and Ys'j are the i-th and j-th row of Y, and Y,', respectively. Additionally, H; E xd, denotes a compatibility matrix that matches the dimension of two different node embeddings (i.e., ys: and ys via linear transformation for edge type t. Incidentally, the Hy-dependent term resembles a form of score function often used to learn knowledge graph embeddings [13
The energy function 8 exhibits the following two characteristics:
The first term prefers that the embedding Y, should close to that from f(X;; W.).
The second term prefers that for two nodes of type and s' connection by an edge type the embeddings Y.i H,, and ys'j should be relatively close to one another.
Even when the dimensions of two embeddings the same (i.e., d, d. H, still serves important role in allowing the embeddings to lie within different subspaces to obtain compatibility.
3.3 Analysis and Derivation of Corresponding Descent Steps
The objective (8) can be written matrix form follows:
(Y Σ
YY. f(X.; W.)l
Σ Σ uT((HHH,) " (Y,HH. -(. A,Y, + TAY,,,,)YYY... +Y,DD. xo)
(9)
E'ES TET
where A, denotes the adjacency matrix for edge type t, and D,t denotes the type-t degree matrix of type-s nodes.
The closed-form optimal point of is provided by the following result; the Supplementary Materials for the proof.
Lemma 3.1. The unique solution Y (W,H) minimizing
9
satisfies
vec(Y Y (W, H)) (I A(Q - P D)) (P+)))''oecll
(10)
where vec(B) [bT, ..... TT for matrix B, H A (H.)ee denotes the set of all compatibility matrices, and we have defined the matrices P, Q and as
P
PIJs
P
Pee
Σ (H. H.,,) Ar)
Pis
...
P[S||S]
P
TET
Q
+ Qui ΣV (H. & Dat,,D IIDD..
VESIETT..
Here & denotes the Kronecker product, + A, Biag(Ai,.. A,) denotes direct sum of square matrices Aty..., An and D. a Eves EieT,, Dst represent a sum of degree matrices over all node types s' E S and all edge types E Test
5

---

We can interpret this solution as transforming f (X; into an embedding that has both local and global information of the graph structure [39] Therefore, this can be treated as an appropriate graph-aware embedding for carrying out a downstream task such as node classification.
However, for practically-sized graph data, computing the inverse (I X(Q P + D)) in 10 is prohibitively expensive, To resolve this issue, similar to the homogeneous case we instead apply gradient descent to ( over Y, for all node types E S approximate Y (W,, H). Starting from the initial point Y ; W.) )) for each E S, several steps of gradient descent are performed to obtain an appropriate embedding for carrying out downstream task. Since all the intermediate steps of gradient descent are differentiable features w.r.t W and H, we can just plug-in the result of performing K' iterations of gradient descent, Y!?), into the 18
In order to get Y!K) using gradient descent, we first compute the gradient of 9 w.I.t
Vv, (Y) = (I XD,)Y, f(X,; + Σ Σ (H,HY.(H.H))))) - A.Y. (H; + He,.))
+
d'ESIET,,
(11)
Note that if the condition number of [1] is large, the convergence speed of gradient descent can be slow (21). To reduce the convergence time effectively, we use the Jacobi preconditioning technique
01. rescaling the gradient step using D- A (I AD.)-1 Then iteration k + of gradient descent
on Y, can be computed as
(1+1)=
(1 a) Y +aD, (
Σ E s'ES TET,.
(A.Y! (H, + Ho... Da: (
(a)
(b)
(c)
(d)
(12)
where denotes step size. The above expression be considered forward propagation rule for the k-th layer in a GNN model, and when re-expressed on a per-node basis shares several commonalities with the node-wis updates of R-GCN (2)), commonly-used heterogeneous GNN variant that is motivated in a completely different fashion. In (12), the terms (a), (b), (c), and (d) can be interpreted follows:
Terms (a) (b) Each term can be treated as skip connection from the previous layer Y!!)
and the imput features f(X,; W.), respectively. Unlike some previous work [15] 18] that adopted
skip-connections as a heuristic solution to resolve the oversmoothing problem [6 22 similar to
unfolded GNNs in the homogeneous case B 36), the skip connections in our expression are naturally derived from the gradient descent step for minimizing the energy function.
Term (c) This term accumulates the propagated feature vectors of neighboring nodes. The matrix H. + Him. serves as edge-type specific transformations in bi-directional way. Note that all matrices in H. are shared across all propagation layers, and as result, the number of parameters does not increase as the propagation proceeds.
Term (d) This term a self-loop transformation that depends on the edge type t. The matrix H,H, introduces an edge-type specific transformation in uni-directional way, and the matrix D.. reflects the strength of the self-loop transformation relying on edge type f. Note that, unlike prior work edge-type specific degree and transformation matrices, and these also shared across all GNN layers.
18 29
We reiterate that all components of 12 have naturally emerged from the gradient descent procedure for minimizing the underlying energy function. And in terms convergence, we have the following result; again, the proof is deferred to the Supplementary Materials:
Theorem 3.2. The iterations (22) are guaranteed to monotonically converge to the unique global
minimum of 8 provided that
2 2Xd
a < 1 + X(dmin + Tmax.
(13)
"This form of the gradient implicitly assumes that for every directed edge there is an inverse edge pointing in the opposite direction. However, the more general case can also be handled with appropriate modification of the gradient expression.

---

where dmin is the minimum diagonal element of D and max is maximum eigenvalue of matrix (Q P).
Time Complexity Although seemingly complicated, the time complexity of executing (2) once is O + ndd), where d = [d, S). In this way the complexity is on the same level as R-GCN 29), and can be efficiently implemented through off-the-shelf sparse matrix libraries.
3.4 Generalization to Nonlinear Activations
In various message-passing GNNs 99
14
18 29) node-wise ReLU activations are applied to the
intermediate features during the forward propagation step. our setting, by adding additional regu- larization constraints
original energy function and using proximal gradient descent methods 24), such nonlinear activations can also be naturally derived without any heuristic modifications on propagation step. Similar ideas have been applied to simpler homogeneous GNN models [36]. More specifically, let Φ. Rd, + R be an arbitrary convex function of node embeddings. Then our optimization objective can be modified the minimization of
(Y
ΣΣ (1
(14)
#ES i=1
Instead of directly minimizing 14 via vanilla gradient descent, we employ the proximal gradient descent (PGD) method (24) as follows. We first denote the relevant proximal operator as
prox (v) arg min "/
l|
212 +ly).
(15)
Then PGD iteratively minimizes (4) by computing
(k++)) == Y aD, L Ex
(16)
Y (k+1) == prox (Y(++1)).
(17)
For our purposes, we set (3) - Σde I=o 0J, which Iso an indicator function that assigns an infinite penalty to any yi < O. In this way, the corresponding proximal operator be- comes prox,, (v) = ReLU(v) max 0, v), in which the maximum is applied each dimension independently.
Algorithm HALO algorithm
Require: G: Heterogeneous graph dataset
Require: a: Hyperparamters, K Number of unfolding steps, E: Number of epochs Randomly initialize H, E H. W. W, and 0₂ E θ
for 1..... E.E do
Set Y = f(X.; W. for all s SES
for k 0................... K 1 do
Compute 16) using 12). and 17) get (++1) for each s end for
Compute (0, W, H) using !!K)
Optimize all W. E W. H. H. and 0, E θ using SGD.
end for
3.5
The Overall Algorithm
After performing K iterations of 12 with the proximal operator in 17 we obtain (W)) (W,,), H.). and analogous to (T. the meta-loss function for downstream tasks becomes
n''
(, = Σ
D(o
(K) (W,, (W,H): 0,), f.ss:
(18)
"ES'
where S' denotes a set of labeled node types, 9s : Rd. g denotes a node-wise function
parameterized by 0,, θ {0.) (0.)ess denotes of parameters of gs- and t,, R is a ground- truth node-wise target Finally, we optimize (18) over W, H. and O via typical iterative optimization method such as stochastic gradient descent (SGD).
Note that w.l.o.g..
that only the first n¹ < n. type-s nodes have labels for training.
7

---

Table 1: Results on HGB (left) and knowledge graphs (right). The results are averaged over 5 runs.
Dalasel
DBLP
IMDB
ACM
Freehuse
Avg.
Dalasel
AIFB
MUTAG
BGS
Avg.
Metric
Accuracy (%)
Metric
Accuracy (%)
R-GCN 29
92.07
62.05
91.41
58.33
75.97
Feat 26
53.55
77.94
72.41
66.66
68,
HAN
92.05
75.56
WL
80.88
87.37
83.75
HGT 1T
93.49
67.20
91.00
60.51
78.05
RDF2Ved 28
88.88
67.20
87.24
88.33
82.91
Simple-HGNI [88
94.46
67.36
93.35
80.37
R-GCN
95.83
73.33
83.10
89.29
85.39
HAALO
96.30
76.20
94.33
62.06
82.22
HALO
96.11
86.17
93.10
90.20
91.40
We dub the model obtained by the above procedure as HALO for Heterogeneous Architecture Leveraging Optimization, and Algorithm summarizes the overall process.
4 Experiments
We evaluate HALO on node entity classification tasks from the heterogeneous graph benchmark (HGB) [18] as well as several knowledge graph datasets [29]. and compare with state-of state-of-the-ar oft baselines. Furthermore, and we carry out additional empirical analyses and ablations.
later evaluate w.r.t. ZooBP under settings that permit direct comparisons,
4.1 Node Classification Benchmark Datasets
The dataset descriptions are follows:
HGB contains node classification datasets, which is our focus herein. These include: DBLP, IMDB, ACM, and Freebase. DBLP and ACM are citation networks, IMDB is movie information graph, and Freebase a large knowledge graph. Except for Freebase, all datasets have node features (for Freebase, we set the node types as the node features). For more details, please refer to [8]
The knowledge graph benchmarking proposed in [29 is composed of 4 datasets: AIFB, MUTAG, BGS, AM. The task is to classify the entities of target nodes. AIFB and MUTAG are relatively small-scale knowledge graphs, while BGS and AM are larger scale, e.g., the AM dataset has more
than IM entities. For more details, please refer to [29].
In every experiment, we chose fX,; W 8) and (3,, (.) as linear functions for all S, and when the dimension dgu is different across node types, fX,,; Ws) is naturally constructed to align the dimensions of each feature. All models and experiments were implemented using PyTorch (25] and
the Deep Graph Library (DGL) 32). For the details on the hyperparameters and other experimental settings, please see the Supplementary Materials.
Table shows the results on node classification tasks, where the best performance highlighted in bold. The left side of the table shows the results HGB, including baselines R-GCN [29] HAN [3), HGT [1] and Simple-HGN [8]. The right side shows the analogous results on the knowledge graph datasets, where also include comparisons with R-GCN [29], hand-designed feature extractors
(Feat) [26]. Weisfeiler-Lehme each table represents the average of the results across the four datasets. In the table, HALO achieves the highest average accuracy, and outperforms the baselines on 7 of 8 datasets.
kernels (WL), and RDF2Vec embeddings (28). The Avg. column in
4.2 Comparison with ZooBP
We next compare HALO with ZooBP (6), which provided motivation for our of compatibility matrices and represents a natural baseline to evaluate against when possible. ZooBP is a belief propagation method for heterogeneous graphs, and therefore, the number of classes per node type should be pre-defined for every node. Consequently, running ZooBP on HGB or the knowledge graph datasets in which only a single node type has labels (and therefore defined classes), e.g., DBLP, not directly feasible. To that end, to compare with ZooBP, we modified the DBLP and Academic BT datasets so that all node types have labels. Namely, in both graphs, we removed the "Term" and "Venue" node types and only utilized the "Paper" and "Author" node types. Moreover, since the original datasets only have the class labels for the "Author" type, we set the class labels for the "Paper" type as the corresponding "Venue" of the paper. Thus, the 20 and 18 Venues (specified in the Supplementary Materials) become the class labels for the "Paper" nodes DBLP and Academic, respectively. Furthermore, we also considered simpler setting in which the number of class labels for the "Paper" nodes (denoted as k) is four, by using the 4 categories (i.e. MI., DB, DM, TR) of the Venues as the labels for the nodes. We note the number of class labels for the "Author" nodes
8

---

Table 2: Comparison with ZooBP
Datasct
DBL.Preduced
Academie-roduced
Metric
Accuracy (%)
Node types
Author Paper / All
Author / Paper / All
# Classes of "Paper"
k= 4
k == 20
==4
k = 18
[on
63.00 63.60 63..47
62.007 62.00/19.53/288,
19.53728..77
79.63/ 0333773.377550 75.97
81,767 16.08/44.9 97.81 88473/ 90.18
Ours
76.50/ / 64.13 / 66.79
78.50 33.33 43.00
96.63 / 93.89 94,93
was always set to 4 (corresponding to the Venue categories), and the resulting datasets are denoted as DBe.Prrddueed and Academic-reduced, respectively
For choosing the fixed compatibility matrix required by ZooBP (as a label propagation method, ZooBP has no mechanism for learming). when & 4, we set H; = x = where 1, is a k x k square matrix in which all elements are 1. When A = 18 or k = 20, we set the matrix considering the correlation between the "Author" and "Paper" labels. For example, for the "AAAI" Paper class, we set the matrix element value corresponding to the "ML" Author class higher than other Author classes, since "AAAI" venue and "ML" category are highly correlated. For further details about reducing the number of classes for the "Paper" nodes, setting the compatibility matrices, and the exact compatibility matrices that we used, please refer to the Supplementary Materials. Moreover, for setting the output layer Os (ys, 0.) in our case, we set 9s (ys: 0,) as an identity function, i.e.,
0.- = I, and we treat y, as a score vector on each class.
Table 2 shows the results of ZooBP and HALO on the DBLP-bipartite and Academic datasets as described above. We also report the itemized classification accuracy on "Author" and "Paper" node types. From these results, we observe that HALO outperforms ZooBP in all settings. For large k. the gap is generally more significant. The main reason for this phenomenon is that while HALO utilizes both node features and trainable compatibility matrices, ZooBP only uses training labels with fixed compatibility matrices. To the best of our knowledge, HALO the only method designed exploit a trainable compatibility matrix.
Accuracy
Energy functian (x10*)
Fixed
HALO w/o
Methods
HALOO
3.0-
39-
Meuri
Accuracy (56
80
6.5
DBLP
96.30
95.40
93.90
MUTAG
86.17
77.94
5.0
BGS
93.10
86.90
86.17
Training Validation
5.3
40
60
60
100
90
40
50
50
70
60
90
100
Propagation Steps
Propagation Steps
Table 3: Ablation study
Figure 1:
Accuracy and energy function value versus the number of propagation steps on ACM.
4.3 Further HALO Analysis
In this section, we first carry out an ablation study to assess the importance of learnable compatibility matrices H, E H. well as the value of non-linear graph propagation operators instantiated through PGD. Later, we also show the accuracy and energy values of HALO versus the number of propagation steps to demonstrate convergence of the lower-level HALO optimization process, which naturally mitigates oversmoothing.
Ablation Study To examine the effectiveness of trainable compatibility matrices, we enforce H. I for all t, using the same embedding dimension across all node types. Other training conditions remain unchanged. Table Bable:commprrs the resulting accuracy DBLP, MUTAG, and BGS datasets, where we observe that HALO performs significantly better, especially on MUTAG and BGS. These results show that using trainable compatibility matrices quantify ambiguous rel relationships between neighboring nodes can be an effective strategy when performing node classification with heterogeneous graphs. Note that even when the embedding dimensions are same (as in these experiments), there can still be
'The graph in DBLP-reduce is a bipartite graph since the edges exist only between "Paper" and "Author" nodes, while the graph in Academic-ee medeeee also has edges between the "Paper" nodes.

---

relation sprocifcc mismatches between different node types that exhibit heterophily characteristics, which trainable compatibility matrices can help resolve.
Turning to the non-linear activations, we also re-run the HALO experiments with (17) removed from the algorithm, meaning PGD defaults to regular SGD as no proximal step is used. As shown in Table
3 HALO without PGD performs considerably worse, implying that the inclusion of the additional non-linear activation leads better generalization.
Results Varying Propagation Steps Figure F shows the results of HALO with various numbers of propagation steps. Specifically, the left plot shows the training and validation accuracy the ACM dataset, and the right plot shows the energy function value ( during the unfolding steps. Notably, when increase the number of propagation steps with suitable a, both training and validation accuracies are gradually increasing, and the energy function value steadily decreasing as expected. These results suggest that the gradient descent steps gradually decrease the low-level energy function, the corresponding node embeddings Y, that emerge are effective in performing the downstream classification task. We have also repeated this experiment using the AIFB dataset and the overall trend is same; for more details please refer to the Supplementary Materials.
Conclusion
In this paper, we have proposed HALO, novel HGNN architecture derived from the minimiza- tion steps of a relation-aware energy function, the latter consisting of trainable relation relation-dependent compatibility matrices that can resolve the mismatch between two different node types. Because of intrinsic properties of this unfolding framework, HALO is naturally robust to oversmoothing problems, and outperforms SOTA models on various benchmark datasets. For future work, extending HALO spatio-temporal graphs in which the node and edge information change continuously can be considered.
Acknowledgments
This work was supported in part by the New Faculty Startup Fund from Scoul National University, NRF grants 2RAF22021R12222008888 0-01343, No. N0.2001--002006 No. No.2022-0-00959 No. 20.22022-00011 13] funded by the Korean government, and SNU-NAVER Hyperscale AI Center. We also thank Zheng Zhang from Amazon Web Services for suggesting interesting connections with techniques for learning knowledge graph embeddings.
and IITP grants [No.2021- NRF-2021M33D22000D4777
References
[1] Owe Axelsson. Iterative solution methods. Cambridge university press, 1996.
[2] Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. Representation learning for attributed multiplex heterogeneous network. In International Conference on Knowledge Discovery and Data mining, 2019.
[3] Jiuhai Chen, Jonas Mueller, Vassilis N Ioannidis, Soji Adeshina, Yangkun Wang, Tom Goldstein, and David Wipf. Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features. In International Conference on Learning Representations, 2021.
[4] Gerben Klaas Dirk De Vries and Steven De Rooij. Substructure counting graph kernels for machine learning from RDF data. Journal of Web Semantics, 2015.
[5] Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. Metapath2vec: Scalable representation learning for heterogeneous networks. In International Conference on Knowledge Discovery and Data mining, 2017.
[6] Dhivya Eswaran, Stephan Günnemann, Christos Faloutsos, Disha Makhija, and Mohit Kumar. ZooBP: Belief propagation for heterogeneous networks. In International Conference on Very Large Databases, 2017.
[7] Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. MAGNN: Metapath aggregated graph neural network for heterogeneous graph embedding. In The Web Conference, 2020.
10

---

[8] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Combining neural networks with personalized pagerank for classification on graphs. In International Conference on Learning Representations, 2019.
[9] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems, 2017.
[10] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. GPT-GNN: Genera- tive pre-training of graph neural networks. In International Conference on Knowledge Discovery and Data mining, 2020.
[11] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In The Web Conference, 2020.
[12] Vassilis N Ioannidis, Meng Ma, Athanasios N Nikolakopoulos, Georgios B Giannakis, and Daniel Romero. Kernel-based inference of functions graphs. In Adaptive Learning Methods for Nonlinear System Modeling. 2018.
[13] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[14] Thomas N Kipf and Max Welling. Sumi-super classification with graph convolutional networks. In International Conference Learning Representations, 2017.
[15] Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. DeepGCNs: Can GCNs go as deep as CNNs? In International Conference on Computer Vision, 2019.
[16] Qimai Li, Zhichao Han, and Xiao-ming Wu. Deeper insights into graph convolutional networks for semi-supprvvised learning. AAAI Conference on Artificial Intelligence, 32, Apr. 2018.
[17] Xiang Li, Danhao Ding, Ben Kao, Yizhou Sun, and Nikos Mamoulis. Leveraging meta- meta-path path contexts for classification in heterogeneous information networks. In International Conference on Data Engineering, 2021.
[18] Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang. Are we really making much progress? Revisiting. henchmarking and refining heterogeneous graph neural networks. In International Conference on Knowledge Discovery and Data mining, 2021.
[19] Yao Ma. Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal demoising. arXiv preprint arXiv:2010.0177 2020.
[20] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011.
[21] Jorge Nocedal and Stephen J. Wright. Numerical optimization. Springer, 2006.
[22] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference Learning Representations, 2020.
[23] Xuran Pan, Shiji Song, and Gao Huang. A unified framework for convolution-base based graph neural networks, 2021.
[24] Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in Optimization, 2014.
[25] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems. 2019.
[26] Heiko Paulheim and Johannes Fümkranz. Unsupervised generation of data mining features from linked open data. In International Conference on Web Intelligence, Mining and Semantics, 2012.
[27] Judea Pearl. Reverend bayes on inference engines: A distributed hierarchical approach. In Probabilistic and Causal Inference: The Works of Judea Pearl. 2022.
11

---

[28]
Petar Ristoski and Heiko Paulheim. RDF2 RDF graph embeddings for data mining. In International Semantic Web Conference, 2016.
[29] Michael Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, 2018.
[30] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. SeiisfellerLehmane graph kernels. Journal of Machine Learning Research, 2011.
[31] Jie Tang. Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnetminer: extraction and mining of academic social networks. In International Conference on Knowledge Discovery and Data mining, 2008.
[32] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, tnehly-performant package for graph neural networks. arXiv preprint arXiv: 0130011112000 2019.
[33] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heteroge- neous graph attention network. In The World Wide Web Conference, 2019.
[34] Yuto Yamaguchi, Christos Faloutsos, and Hiroyuki Kitagawa. CAMLP: Confidence-aware modulated label propagation. International Conference on Data Mining, 2016.
[35] Yongyi Yang. Zengfeng Huang, and David Wipf. Transformers from an optimization perspective. In Advances in Neural Information Processing Systems, 2022.
[36] Yongyi Yang. Tang Liu, Yangkun Wang, Jinjing Zhou, Quan Gan, Zhewei Wei, Zheng Zhang, Zengfeng Huang, and David Wipf. Graph neural networks inspired by classical iterative algorithms. International Conference on Machine Learning, 2021.
[37] Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. Advances in Neural Information Processing Systems, 2019.
[38] Hongwei Zhang, Tijin Yan, Zenjun Xie, Yuanqing Xia, and Yuan Zhang. Revisiting graph convolutional network on semi-supervised node classification from an optimization perspective. CoRR, 2020.
[39] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernaard Schölkopf. Learning with local and global consistency. In Advances in Neural Information Processing Systems, 2003.
[40] Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. arXiv preprint 2rX1v:2101.11859 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Supplementary
Materials.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see Supplementary Materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] Please see Supplementary Materials.
(b) Did you include complete proofs of all theoretical results? [Yes] Please see Supple- mentary Materials.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] Codes can be submitted after acceptance.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please see Supplementary Materials.
(c) Did report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Please see Supplementary Materials.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Please Supplementary Materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "cEE4zITg_X9", "reviewer": "Reviewer_zy2F", "summary": "This paper proposes a systematical way of constructing energy functions for heterogenous graphs with multiple edge types and node types. This approach is justified from both the theoretical perspective and the experiments.\n", "questions": "See above.", "limitations": "See above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n* The presentation is very clear. The authors analyzed the drawbacks of direct extention of graph regularizer from single-edge type graph and proposed a term that considers the inter-edge-type correlation as a regularizer with rigorous theory analysis.\n* The results of combining proximal gradient and nice trick of gradient step rescaling (L209) makes the overall procedure sound and valid.\n* The paper achieves state-of-the-art results in multiple datasets.\n\n### Weakness:\n* I only have a small presentation-level comment. It would be better if the authors can move the comparing part of their proposed $y_{si}H_t-y_{s'j}$ in Eq (8) to the introduction/abstract section to let the readers know the intuition that cross-edge-type correlation is important to heterogenous graph learning. Also it can be better if the authors can visualize what the learned $M_t$ is to justify that it's not a diagonal matrix.\n\n#### Minor:\n3. L139 (ii) \"the loss can be equivalently expressed with symmetric weights\" is unclear to me.\n4. L143 (iv) the expression that \"symmetrical w.r.t. order of nodes\" is confusing. $M_t$ can be asymmetrical. Can the authors explain more on this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "S-BMij_s-4", "reviewer": "Reviewer_Dn37", "summary": "This paper propose a new heterogeneous GNN motivated by minimizing relation-aware energy function. In short, it tries to make connected nodes share similar labels, with a relation-dependent compatibility matrices to resolve the mistmatch between different node types. The proposed method achieves improvement on heteogenous graph benchmarks.", "questions": "1. The energy regularization seems to be similar to label propagation, and recently there exist several attempts to combine label propagation with GNN, such as [1]. Could the authors discuss their difference?\n\n2. After reading the paper and Alg 1. I think the proposed method is more like a learning framework, and it should be agnostic to different base HGNN architecture (f in alg1)? Could the authors provide more details about it? If it's indeed model-agnostic, could the authors conduct experiments with different base function f?\n\n3. I'm still not pretty clear why the authors choose proximal gradient descent instead of standard optimization. Could the authors provide more discussion or add some ablation study?\n\n[1] Combining Label Propagation and Simple Models Out-performs Graph Neural Networks", "limitations": "The authors claim that limitation is included in supplementary but I didn't find it. I suggest the authors to add some discussion about the usage boundary and potential limitation of this method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. This paper took non-trivial efforts to adopt unfolded optimization techniques (somehow force connected nodes to share similar labels) into heterogeneous GNN. There exist some challenges, such as the label and distribution mismatch across different node types, how to integrate the deterministic label adjustment with the non-linear model (the authors adopt a proximal gradient descent)\n\n2. I appreciate how the authors try to improve the simple form in eq(7) by considering heterophily (not simply considering connected nodes to have similar labels), introducing a projection matrix H, and give a detailed and closed-form solution for the modified method.\n\nWeakness:\n\n1. The proposed method has many design choices, such as a naive regularization in eq (7), standard optimization, etc. Current ablation studies only change projection matrix H. It would be better if the authors could conduct more thorough ablation studies and tell us which component contributes most.\n\n2. This paper has already conducted lots of experiments. However, it would still be good if they could also choose larger datasets such as OGB-MAG and OGBLSC-MAG and report the results. Also, it would be good to also show the model parameter and inference time (or FLOPS).\n\n\n\nPersonally I like this paper, and I think the paper could be improved if the authors could make the experiments more solid and provide more details and discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "RA19qfSihFx", "reviewer": "Reviewer_U2aH", "summary": "This paper studies heterogeneous GNN and proposes a novel HGNN architecture HALO, derived from the minimization steps of a relation-aware energy function. HALO introduces trainable compatibility matrices to solve the mis-matches between two different node types. Experiments on 8 benchmark datasets show the effectiveness of the proposed model.", "questions": "Please see the comments above.", "limitations": "I think the high space complexity of the model is a limitation that needs to be further improved, which is omitted by the authors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1 The paper is well-written and easy to follow.\n\n2 The proposed method HALO is derived by the gradient descent steps for the minimization of a relation-aware energy function, which is more explainable.\n\n3 Experiments on 8 benchmark datasets show the superiority of the proposed method.\n\nWeakness:\n\n1. The proposed method HALO is a direct extension to the model proposed in [32] to heterogeneous graphs. Compared with the model in [32], HALO introduces compatibility matrices to map embeddings of nodes of different types into the same low-dimensional space. Therefore, the proposed model is a bit incremental.\n2. There exist some confusions in the paper. For example, in line 133, the authors write “Eq 5 fail to capture the nuances of heterogeneous data”. I think this is incorrect because in Eq 5, node types are taken as nodes features, and edges types are also considered.\n3. For experiments, I have several concerns. First, for Table 1, I even do not know what the evaluation measure is before I step into the main texts. Further, based on one metric of classification accuracy, the results is somewhat unconvincing. Finally, since alleviating the over-smoothing issue is an important contribution of the proposed model, conducting experiments on more datasets is a necessity, while in the current version, the only results on ACM seem to be cherry-pick.\n4. Some important baselines are missing in this area:\n(1) Representation learning for attributed multiplex heterogeneous network\n(2) Graph Transformer Networks\n(3) Leveraging Meta-path Contexts for Classification in Heterogeneous Information Networks\n5. While 8.9 pages are not a big issue, the blank part is very easy to fill, which can make the paper more compact.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3OkFfSuh6uu", "reviewer": "Reviewer_Sdud", "summary": "This work proposes a novel graph neural network architecture, HALO, for heterogeneous graphs. HALO is derived from the minimization steps of a relation-aware energy function, which addresses the oversmoothing issue in terms of heterogeneous graphs. Extensive experiments conducted on the node classification task in 8 heterogeneous graph datasets show the effectiveness of the proposed method. ", "questions": "See the Strengths and Weaknesses.", "limitations": "The authors didn't address the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The approach to solve the oversmoothing issue in terms of heterogeneous graphs is novel.\n2. Paper is well-written and easy to follow.\n3. The proposed model significantly outperforms other HGNN baselines in several datasets (e.g., IMDB) for node classification.\n\nWeaknesses\n1. It would be good if there was a comparison with other works that addressed the oversmoothig issue in homogeneous graphs. Then, the oversmoothing issue in heterogeneous graphs can be experimentally understood more.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept