HGDL: Heterogeneous Graph Label Distribution Learning
Authors: Yufei Jin
Heng Lian
Yi He
Xingquan Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Heterogeneous Graph
Graph Representation Learning
Label Distribution Learning
Primary area: graph_neural_networks

Abstract: Label Distribution Learning (LDL) has been extensively studied in IID data applications such as computer vision, thanks to its more generic setting over single-label and multi-label classification. 
This paper advances LDL into graph domains and aims to tackle a novel and fundamental
heterogeneous graph label distribution learning (HGDL) problem.
We argue that 
the graph heterogeneity reflected on node types, node attributes, and  neighborhood structures can 
impose significant challenges for generalizing 
LDL onto graphs. 
To address the challenges, we propose a new 
learning framework with two key components: 
1) proactive graph topology homogenization, 
and 2) topology and content consistency-aware graph transformer. 
Specifically, 
the former learns optimal information aggregation between meta-paths, so that the node
heterogeneity can be proactively addressed prior to the succeeding embedding learning; the latter leverages an attention mechanism to learn consistency between meta-path and node attributes, allowing network topology and nodal attributes to be equally emphasized during the label distribution learning. By using KL-divergence and additional constraints, \method~delivers 
an end-to-end solution for learning and predicting label distribution for nodes. 
Both theoretical and empirical studies substantiate 
the effectiveness of our HGDL approach.
Our code and datasets are available at https://github.com/Listener-Watcher/HGDL.

HGDL: Heterogeneous Graph Label Distribution Learning
Yufei Jin', Heng Lian', Yi He,, Xingquan Zhu Dept.. of Elec. Eng. & Computer Sci., Florida Atlantic University, Boca Raton, FL 33431, USA Dept. of Data Science, William & Mary, Williamsburg, VA 23185, USA yjin2021 (faoo21@fauu.om edu; hlian01 @wm. edu; yihe@um.... edu; xzhu3@fau edu
Abstract
Label Distribution Learning (LDL) has been extensively studied in IID data appli- cations such computer vision, thanks to its more generic setting over single-label and multi-label classification. This paper advances LDL into graph domains and aims to tackle novel and fundamental heterogeneous gruph label distribution learning (HGDL) problem. We argue that the graph heterogeneity reflected on node types, node attributes, and neighborhood structures can impose significant chal- lenges for generalizing LDL onto graphs. To address the challenges, we propose a new learning framework with two key components: 1) proactive graph topology homogenization, and 2) topology and content consistency-awart graph transformer. Specifically, the former learns optimal information aggregation between meta- pattappahss so that the node heterogeneity can proactively addressed prior to the succeeding embedding learning; the latter leverages attention mechanism learn consis- tency between meta-path and node attributes, allowing network topology and nodal attributes to be equally emphasized during the label distribution learning. By using Kl-divergence and additional constraints, HGDL delivers an end-to-end solution for learning and predicting label distribution for nodes. Both theoretical and empirical studies substantiate the effectiveness of our HGDL approach. Our code and datasets are available at http://ggghhu...mmi.toeetteee http:///issstteeeeectctceee////a
Introduction
Definite supervision signals are often postulated in learning settings yet, data generated from the real world tend to present inherent ambiguity, imposing challenges on assertive classifiers that predict instances into single or multiple classes. Label Distribution Learning (LDL) [5] 6
has emerged to navigate label ambiguity by pursuing mapping from instances to their class distributions. Each distribution quantifies the descriptive degrees of various classes given a specific instance.
focus on independent and identically distributed (IID) data, such as images or texts, which do not generalize well on graphs. In fact, the topological structure underlying instances may provide invaluable information for label distribution learning. For example, in the task of urban planning, recent learning models have been employed to predict the point of interests (POIs) of local regions by providing the regional distributions over all POIs. which lends finer- finer-granular delineation of urban regional functionality instead of single- or multi- cluls classification. To wit, for region that mixes four POIs (classes): housing, healthcare, education and worship, unlike other models assertively classify it into one multiple POI(s), LDL model can provide insights of the functional degrees of all four POIs in this region, as shown in Figure T Nevertheless, existing LDL. studies overlook the
However, the existing LDL studies mainly 10
6 7
[2] [3] 14 15] LDL can further extend this task
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

urban topology, which can be rendered from, e.g., the taxi services across regions 22, missing out critical city traffic patterns that are highly correlated with regional functionalittes. For instance, the regions with balanced POI distributions (e.g., R2 and R3) are less likely to form connections with other nodes compared to regions heavily skewed towards a single class (e.g., R4), as their residents enjoy fewer needs to travel to other regions for services such as education and healthcare.
In this paper, we aim to enable and generalize the label distribution learning paradigm in networked data. Two technical challenges confront our study. First, real-world graphs are mostly heterogeneous, comprising diverse types of nodes for better expressiveness. Graph heterogeneity complicates the message-passing between nodes of a spe- cific type (e.g., residence), as the label distributions of those nodes are influenced by their neighboring nodes that may vary in terms of types, content, and topological fea- tures. Simply leveraging node embeddings generated from message-passing for LDL will thus not work well [16] [7]. To aid, although meta-path -att aggregation [8] is seemingly viable, necessitates extensive domain knowledge and ex- pertise to craft meta-paths for each node type with respect to their label distributions; given the combinatorial num- ber of possible meta-paths in large heterogeneous graphs searching for the optimal meta-path for LDL. is costly, laborious, and rime-demanding.
--- cI 12 44 residence
residence
C3
C4 3
service
service
Leisure
R
residence
service
Transit
C4
residence
Classes (POls)
CI Housing
Leisure
C2 Healthcare
cl C2 94
C3 Education
c4 Worship
Figure 1:
Motivating example of HGDL
study, where each node is a local urban re- gion [ and edges represent taxi services commuting between regions. Heterogeneous node types indicate disparate land use, includ- ing residence (R), service (S), leisure (L), and transit (T), among which R nodes are of our interest. Colored R nodes are with ground truth, which delineate their distributions over multiple point-of of of-interests (POIs), and each POI is decmed as a class/label. Our HGDL problem is to predict the label distribution of uncolored R nodes, enabling a precise delin- eation of regional urban functionality.
Second, graph topology and nodal features may suggest inconsistent label distributions, where nodes sharing simi- lar contents are positioned far apart on the graph topology. The inconsistency is furthered in heterogeneous graphs, where nodes of the same type often connect through other intermediary types, resulting in substantial topological distances between them. Unlike traditional LDL that fo- cuses instance vectors only, effective LDI. model graphs require harmonizing nodal contents with topolog- ical structures for unified representation. The impact of distantly positioned nodes within a graph is substantially diminished, consequently steering the LDL model to pri- oritize individual nodal vectors, leading to compromised node representations in which the informative patterns embedded in their neighborhood structures are overlooked. Such patterns, which may significantly enhance label distribution predictions as illustrated in Figure I are neglected, undermining the LDL model effectiveness.
To overcome the challenges, we propose a new learning framework, coined Heterogeneous Graph label Distribution Learning (HGDL). Specifically, tame the graph heterogeneity, HGDL learns the optimal graph topology of the target nodes from multiple homogeneous structures searched with various meta-path schemes through a tailored attention mechanism. The node embeddings are then generated by harmonizing the nodal features and the learned meta-path graph using a transformer architecture. A joint optimization objective is crafted based on the distance between true and predicted label distributions of the target nodes from their resultant embeddings, which unifies the learning of meta-path graph topology and the feature-topology harmonization function in an end-to-end fashion.
A key innovation of HGDL is that it changes existing heterogencous graph learning paradigm from reactive (meaning that aggregation of different meta-paths done after embedding learning from individual meta-path), to be proactive (meaning that aggregation are done before embedding learning). Combined with attention and transformer mechanisms to adjust individual meta- meta-paths' interplay, and align with nodal features, HGDL deliver significantly better performance over alternatives. Our theoretical analysis assures that HGDL outperforms that of using an arbitrary meta-path graph, and HGDL's topology and feature consistency learning sparsifies network connectivity, intermediately encouraging tightens the error-bound, resulting in better model generalization.
Specific contributions of this paper are as follows:

---

1. This study pioneers the exploration of LDL problem in heterogeneous graphs. The learning problem enjoys practical implications such for urban functionality delineation (presented in Sec T.T) and, to our knowledge, has not yet been explored by any contemporary research.
2. We propose an end-to-end HGDL learning approach to jointly learn an optimal meta-path graph topology and align it with nodal features for consistent
pessage-passingg Our ap- proach is surprisingly simple yet effective, with its performance evidenced by theoretical underpinnings. Our approach and its analysis are presented in Sections and a respectively. 3. Empirical study has been carried out over five graph datasets that span domains of bio- medicine, scholarly network, business network, and urban planning. Experimental results substantiate the effectiveness of our approach over rival models, documented in Section 6
2 Related Work
Label Distribution Learning (LDL)) strives learn mapping from input distribution that
profiles the descriptive degrees of classes associated with it [5] 10 6 [ Existing LDL methods fall into three categories, namely, problem transformation (PT), algorithm adaption (AA), and specialized algorithm (SA). PT methods transform LDL as multiple single-label learning tasks, using with label probabilities (19), and AA approaches revise mainstream learning algorithm to fit the LDL loss. SA algorithms are most commonly used because LDL learning is driven by new algorithm designs. Label correlation has been found benefit the label distribution learning, where approaches were proposed to encode label correlation to a distance to measure the similarity of any two labels [6]. Later. low-rank approximation is used to construct label correlation matrix to capture the global label correlations
7 Instead of exploring common features for all labels, label-specific features [IO] for each label are used to enhance the L.DI. model. Exploring feature-label and label-label correlation [ has recently been studied in generalizable label distribution learning for cross domain learning. A Gaussian label distribution learning method [11 employs a parametric model underpinned by an optimization strategy assessing KL-divergence distance between Gaussian distributions, followed by regression loss to normalize the KL-divergence distance. Noticing the difficulty to obtain ground- label distributions, Label Enhancement [20] is commonly used to recover label distributions from logical labels. Our research further push LDL to be generalized onto heterogeneous graphs, which have been overlooked by existing research. Although a recent study [21] explored using LDL in topological spaces, it focused on homogeneous graphs only and cannot work in the setting of more than one node type. Thus, the studied problem in 21 and its challenges differ from ours.
Heterogeneous Graph Neural Networks have drawn extensive attention in graph learning [66 17 2 18 23 24), because the graph heterogeneity imposes considerable challenge to model the interplay among various node types, features, labels, and network topology. Using meta-path to aggregate information from different types of nodes/edges is common approach for heterogeneous graph learning. HetGNN [8] designs graph neural networks to encode features for each type of neighbors and then aggregates neighbors' representation with respect to different types. This provides a way for GNN to deal with heterogeneous graph structures and node attributes. HAN [25) introduces attention mechanisms to heterogeneous graph learning, where attentions are applied to embedding features learned from homogeneous networks, each created from meta-path. By doing so, attentions serve as a weighting mechanism automatically determining the importance of each meta- meta-path for learning. Using transformers for heterogeneous networks has also been investigated recently. For example. HGT (26] designs node- and edge-type dependent parameters to characterize the heterogeneous attention over each edge, allowing this method to learn representations for different types of nodes and edges. SeHGNN (27 proposes a transformer based semantic fusion module, allowing feature fusion from different meta--aths. Our research is fundamentally different from existing work in two aspects: 1) we study LDL learning for heterogeneous networks, and 2) we propose new way to aggregate and align information for heterogeneous network.
3 Preliminaries
Notations. A heterogeneous graph is denoted by G XVY,, associated with a node type mapping Φ V + T and an edge type mapping ş E + T", with T and T the predefined and finite sets of nodes and edges, respectively, and [T 2 2. Denote 4, E T as the node type of our interest, and suppose in total n nodes are of this type. Without loss of generality, we have ((v1) = ... = ((v)) l,, and V,_ (vi,..., "n] C V. We deem these n nodes as our target nodes, using a feature matrix X E _axm to describe their nodal contents, where each node contains
3

---

R,
R4
Target Victers
patt Pa:
R.
Fenture Accendion Adjaconcy
Mess
X
Attonbion Scores
R2
81000770
R 00008055005
GONIA
PL -..
Dag 230.0
M
Wi
Trndk
R
jolp towin
4.5
9,3
B[AK] Lapttorn(AA)
4s
PELTLL
Aa
R- S..RR
CONIAG:
PastFin
marto-park
A2
Predicted Distri
montminelles
Dog De--D:tDEL U
Attention Regalarizer
The proposed HGDL framework. Using k meta-paths, the heterogeneous network in ! is converted to k homogeneous meta-path graphs in 2. Topology homogenization in @ proactively aggregates all & meta-path graphs, through learnable weight matrix Wij R for each graph, and finally obtain attention O R across all graphs. Topology and feature consistency-aware graph transformer in ( harmonizes the local and glohal consistencies. The objective function in ( unifies loss and regularization terms guide nodal label distribution learning.
Figure 2:
an mn-dimensional feature vector. A meta-path P is defined as relational sequence in form of 3 f22 ... iS t; " 1,+1.... -t (abbreviated as P (f1 ... fett41 ...4), where (1,44+1) T describes the composite relation between a pair of node types. By defining a meta-path P with same first and last node type as the target node type, i.e. 4 = 4 = 4, we can use P to convert heterogeneous network as a meta-path graph concerning only the target node type, which shall be discussed later Section 4.1
Problem Statement. In our HGDL problem, the goal is to learn predictive mapping h XG Y. where Y [0, 1 a distribution of descriptive labels over q classes. Let Vr, E [0, 1] be the probability that the node v, belongs to the j-th class, we have Σソ= transductive learning regime (28) to allow the ground-truth label distributions known for subset of target nodes Vur C Vi, during training. Our learned mapping h is expected to generalize well so can make accurate prediction on the remaining target nodes Vi, Vir.
1. In this work, we follow
4 HGDL: The Proposed Approach
Overview.
The proposed HGDL approach comprises three key components, as illustrated in Fig- u First, for the target nodes belonging to the node type t, of interest, HGDL generates multiple homogeneous meta-path graphs based on their original locations on the heterogencous graph through meta-paths; the optimal graph topology of this node type then learned from the homogeneous graphs via attention mechanism (Sec 4.7 Second, HGDL learns the embeddings of the target nodes by harmonizing the information sourced from their feature space and the learned optimal topology using a transformer- like neural architecture (Sec 4.2). Third, HGDL minimizes the distance between the predicted and ground-trut label distributions based on the learned node embeddings. We tailor an objective function to unify the three components into one end-to-end optimization problem, in which the optimal graph topology, the harmonization function of the feature and topological information, and the target node label distribution are jointly learned (Sec 4.3
4.1 Optimal Graph Topology Homogenization
For heterogenous graph, by leveraging meta-path idea, multiple different meta-path homogeneous adjacency matrix can be obtained and they can be treated as multiple sources. Graph learning is about exchanging and updating information from neighbor nodes. proper neighbor is therefore important for target node to leamn correct distribution. Given multiple sources, each node will have

---

multiple neighbor sets to choose from for updating. Traditionally, embeddings are learned for all the neighbor sets, and then aggregation over embeddings is learned. Semantics over embeddings are hard to interpret and learn compared with directly learned from different neighbor sets.
To generate a meta-path graph from the original heterogeneous graph, interactions between the meta- path and the heterogeneous graph path are used. Two nodes e and 2/ are connected in the meta meta-path graph, if there exists a path connecting them in the heterogeneous graph, and the path follows the meta- path. Given meta path P- = (t) 4,4i+1 ...4), we say that a path p -(V,,-. Ui, Ui+ls..., v1) in graph G follows the meta-path P. Vi, d(ee) = t. Take graph in Fig. [ as an example. Given meta-path p = (r r),, which defines node type & = r. = t3 = r. Path p = $1. r3) in the heterogeneous graph follows P because all nodes in the path p satisfy 6(22) = t = r, r(((s)) tq ((ra) = t3 = r. Because path (T2, 71)))) follows the meta-path P. an edge is used to connect 12 and r3 in the homogeneous meta-path graph constructed from P. Indeed, each meta-path defines a specific way of information propagation in a heterogeneous network, with resulted meta-path graph capturing unique relationships between target nodes. While defining a single meta-path relatively easy, there often exists many meta-paths; aggregating a variety of meta-path graphs to support the downstream learning task is non-trivial.
After searching the meta-paths connecting the nodes of target type 4, we generate a set of graphs A {Ai,.. .............. in which each adjacency A; E {0, 1]mxx captures the topological structure of homogeneous graph. Denoted by A. g] = 1 means that two target nodes Up and l... with ((pp)) ((E,)) f,, are connected by a meta-path; otherwise, [. ql = 0. Unlike existing studies [25] that yield target node embeddings through reactive meta-path aggregation, where they aggregate local neighborhood information for each A, E to capture ko separate meta-path topologies, our HCDL learns the optimal graph topology from A in proactive fashion. Intuitively. HGDL learns node-level attention scores for various homogeneous graphs A,, to respect the fact that the neighboring nodes may pass messages with varying importance levels in local neighborhoods, while the meta-paths walking across nodes of types other than the target 4, Revisit the motivating example demonstrated in Fig [ where the residence nodes are deemed as the target, the meta- path linking through the service nodes dominates, as the residence nodes are more likely to be linked through service nodes instead of Transit and Leisure nodes. Specifically, the attention scores for the nodes in every A, E A are learned in a GAT regime (2).. defined as:
the i-th meta meta-path
θ softmax (IAAEA (LapNorm
Ā Σ 01:11 LapNorm (1)
denotes Laplacian normalization of A, with D, being A, 's degree matrix and is identity matrix. This term mitigates the imbalanced degree distribution of the meta-path graphs. Denoted by Wi and W are the learnable GAT parameters, where Wi E / maps the meta-path topology of A, onto an f-dimensional semantic space. The operator AA. concatenates all k resultant node embedding matrices from meta-path graphs A, thereby producing an "x lookup matrix, where each node associated with k- dimeenniioa embedding representation, and each latent f-dimension captures the local neighborhood structure of this node. Then, W E f summarize the f-dimensional latent space into one coefficient through convex combination, resulting in attention logits, which are fed into softmax(-) = exp(-)/ Σ, exp(-) to yield the attention matrix θ = [0, 1]m* Take the i-th column vector of θ, denoted by θ|: i] e [0, 1]",, we have the attention scores of target nodes for message- message-passing in the i-th meta- meta-path path graph A, We thus can deem Á in Eq. (1) as the learned optimal graph topology, which is an element- wise linear combination of meta-path topologies with the attention scores broadcast onto all n nodes.
where LapNorm(4A)= (A) R2xn
Note, À is asymmetric, namely Alp, g] A Á(a,z) Vp a q. This is because that the attention score of information aggregation from node Up to Uy may be different from that from fy Up, as their respective local neighborhood topologies naturally differ.
4.2 Local Topology and Global Feature Consistency-Aware Graph Transformer
After obtaining the optimal topology À from all meta-path graphs A,, the next question is how to harmonize it with the feature information to better the target node embeddings. The benefit of such harmonization evident. Revisiting the urban network in Fig I we can envision that a pair of residence nodes tend to be associated with similar embedding vectors because they enjoy two types
5

---

of consistencies: i) local neighborhood topology: their residents tend to travel to similar functional regions for leisure or service purposes, and ii) global feature space: they share similar contents such as house types and number of residing families. These local and global consistencies complement with each other, as the residence nodes having similar contents can be topologically faraway from each other on the urban network, and vice versa.
To harmonize the local and global consistencies, we are inspired by the recent graph transformers (30 and observe that the feature attention suggests a global adjacency matrix, which can be incorporated into the caassage-passing process. We define the graph transformer block follows.
Z = ReXLU()
Àg = LapNorm LsaNorrmsoftmax ((2)))a
where Z R is the node embeddings learned from the optimal graph topology À through local information aggregation, parameterized by W1 Rxxh Denoted by Ag [0, [0.1]=x= is the normalized feature attention adjacency, where and K E R map the embedding matrix Z onto the latent query and key spaces, respectively, such that (ZQ) calculates an n X n node-level attention matrix with respect to the feature space information. Instead of normalizing the attention score by the hidden dimension ht we penalize the feature attention adjacency through an element-wise production @ with the optimal meta-path topology Á. The intuition behind Eq. (2 is that, for each target node, it aggregates information from those neighboring nodes only if their meta-path topology and feature space are both with high attention scores. In addition, Eq. 2 functions similarly to the edge dropout [31]; in lieu of randomly removing edges, we enforce neighbor-set intersection, where the information is only propagated from the neighbors on which the feature space and meta-path topology both agree. Such an intersection sparsity thus lowers the degree of the resultant attention adjacency, thereby uplifting the learning efficacy, which will be substantiated later in See Finally, denoted by H = LeaKy (W2) E RR* are the resultant node embeddings, capturing both local topology and global feature consistencies, which is parameterized by weight W2 E AR
4.3
An End-to-End HGDL Objective Function
Based on the resultant target node embeddings H, we can predict their label distributions as softmax A₂ HW3) E [0, 1jmx9, where Y; ( } is the predicted label distribution of node us. among which Ds. denotes its predicted probability of belonging to the class j. The unified objective of our HGDL framework defined as follows.
min (WDL W., WT. W CRCDL- DKL (Y 7 Ω.
(x = 3id log
Vo.1
Ω
(fli U(L,A),
(3)
Vr.1
where the KL-divergence DKL (Y(l) gauges the discrepancy between the predicted and groundtruth label distributions of the target nodes [32]. The regularization term Ω gauges the distance between the attention scores of the i-th node across k meta-path typologies (denoted by :]; and uniform distribution U1,, k.. We note the minus sign before Ω, thus minimizing this term encourages a larger KL-distance, thereby avoiding the trivial uniform attention distribution (meaning that for each node, the learned attention weights from different meta- paths are encouraged to be as different as possible). y is a tuned parameter to balance the two terms.
5 Analysis
We follow the PAC-Bayes regime to analyze the theoretical performance of our HGDL algorithm by deriving its generalization error bound. We proceed analysis based on the meta-path graph adjacency matrices A = (Ai,.. A& which are searched from the heterogeneous graph G. Throughout the analysis, we assume the nodal feature representations to be residing in an ball of radius B. We argue this a mild assumption, because in implementation we can leverage the batch-norm layers to normalize the resultant node embeddings, such that ||b] ||2 S B, where hę denotes the i-th node's embedding resulted from the j-th hidden layer.

---

Let Lg (f) and (x (R) denote the generalization risk over a graph distribution G and the empirical risk on the target node samples and the learned meta-path topology (X, A), respectively, where (X, A) E G jid
G. We can define:
Lg(h) ( (xxxA)n Bunn (((h(X,)))a)) 8:)l.
L (1) -
where ((.-)) is convex distance metric between two distributions that follows (e(u, l(u, g) s (VP + 1)Ipp 21/2, Vu, P. 9 E R Denoted by h(X, A)fii R2 and y: E [0, 1]" the predicted and ground-truth label distribution of the i-th target node, respectively. Implementing KL-divergence, we have (h(( X, A)( MA) = Ej=1 & X, A) In(h(X. X, J/Vt,)) where the predicted probability that node belongs the j-th class is denoted by h(X, A)(, By analyzing the performance the learned meta-path graph topology Á, we find that:
Theorem 1. Let (A,, (h)) the empirical risk of using the i-th meta-path graph A₁ E Affrr label distribution prediction. With the SGD step-size T, we huve
In k: 17n2
(kx S min [[) A.EA
Remark 1. Theorem indicates that the empirical risk of HGDL is no larger than the minimum empirical risk incurred by training label distribution learner on the optimal meta-path graph, as the error bound the RHS reduces to O(1/n) with constant k and ry. With Stochastic Gradient Descent (SGD) optimizer, larger number of target nodes n will lead to more training updates over them, diminishing the (11/n) bound faster. This finding substantiates the tightness of our meta meta-path learning strategy for the optimal graph topology Á.
Due to page limits, we defer the proof of Theorem and the rest analysis the Supplement. We then analyze the generalization error bound of HGDL and find that:
Theorem 2. Let f E H X G - IRS be l-layer message-passingg neural network with maximum hidden dimension k of which the i-th layer is parameterized by W.. Then for any 8, 7, > 0 and I 1, with probability at least - 5 we have
2(V24 + v2)g
|n/|
Lg (h) -L (x,A) (h) <
Vn
ie m
JE()
max
1/g2/8
+0
((W,) 3 log 5
+ 36
2n
Pm
where D(W.) = 11 ||W:ll2 constant.
El_
(1 /1W.)1)))) bounds the hypothesis space and b is
We remark several key observations from Theorems 1 and 2. First, the generalization capability of the algorithm negatively impacted by higher dimensional label space q. Second, the robustness of HGDL decreases with larger B values, which gauges the magnitude of perturbations thus the inherent high data variance. Third, as the graph neural network architecture becomes deeper (larger () or wider (larger k), the generalization risk increases, suggesting the potential risk of model overfitting, Forth, with larger n, the generalization error bound diminishes, which indicates that the meta-path topology can be better delineated with increased number of target nodes on the graph.
Remark 2.
Remark 3.
By combining Theorems and 2, we observe that the generalization error bound of easy to
HGDL using Á outperforms that of using an arbitrary meta- path graph A;- Further, verify that the maximum degree of Á, denoted by d, is smaller than that of A,, denoted by de, i.e. d < dç. Vi E [k). This rationalizes our graph transformer design in Sec 4.2, where the enforced topology and feature consistency Eq. (2) sparsifies network connectivity, thereby intermediately encourages better model generalization.
7

---

Datasct
node type
# nodes
# edges
# features
labels
6 Experiments
DRUG
40,786
1.737,890
191
28
ACM
20,200
104,976
1,903
DBLP
27,325
148, 246
8,920
YELP
8,052,542
7,905,197
19
6.1 Experiment Setup
URBAN
1,434
42,857
155
10
Table 1: Summary of dataset statistics.
Benchmark Datasets To our best
knowledge, no heterogeneous graph dataset with ground-true label distributions currently exists. To level the comparison study, we prepare five datasets with ground tround-truthh node label distributions using existing heterogeneous graphs, including DBLP [33, ACM (33, YELP, DRUG [34, and URBAN M. Table 1 summarizes the data statistics. A detailed description on the dataset creation and preprocessing, as well as their domain and label semantic meanings, has been deferred to the Supplement B due to space limitation.
Compared Models In total six competitors are identified for comparative study. As no model directly resolving the HGDL problem exists, we employ the state-of-the-art heterogeneous graph neural networks and integrate them KL--divergence loss to learn label distributions of nodes. They include: 1) GCNNL.: A baseline that uses graph constructed from each meta-path to train a vanilla GCN [55) using KI diverreeee as loss function, and reports the best meta-path result; 2) HANKL: This baseline uses HAN (25] to integrate embedding from different meta- meta-paths; and 3) ScHGNNKL: This bascline uses ScHGNN (27). a transformer based approach, to aggregate meta-paths embedding with KL- divergence loss function. For ablation study, we further include three variants reduced from our proposed HGDL method, which include: 4) HGDL-TH² it removes HCDL's topology homogenization (Sec 4.1), which learns embedding from each meta-path graph and reports the best meta-path result; 5)
it uses GCN instead of the transformer (Sec 4.2) to learn embedding to validate HGDL's ' transformer for embedding learning; and 6) HGDLED: it replaces HGDL's topology and feature consistence-aware graph transformer (Sec. 4.2) by using a random edge dropout method 331
Evaluation Metrics To measure the discrepancy between two distributions, i.e., the predicted and true label distributions of target nodes, we identify six metrics: Cosine Distance (COD), Canberra Distance (CAD), Chebyshev Distance (CHD), Clark Distance (CLD); Intersection Score (IND), and Kullback-Leibhe Divergence (KL). Their definitions and calculations are deferred to Supplement B.
Datauet
Model
CAD1
CHD
CLD1
IND1
Wis/Ti//oos
1220)..oo uco
9:21Na.7a
HANgs
0.266 J227
2.155...a
0.632..63
0.579
60/0
No
0 11262.o0
10
0:3672
1/100=
(1640)
0.600
M10
DRUG
HKTD).
(L168
9,179..574
0.217
1,977
0L710
0,392
HGDL
9.371=
0.235
2,004
0.6872100
0,422
4/30
HGDD
02222
0.510
0.340
3,000
0-462
4/300
HGDDLp
0 1
9/602 592
0.259 2000
2.090
0.6811100
0,574
4/30
OCN
0 2m mm
1001
0.312 mm
187
0.842
$11-0
HANg
02121.00 000
0,371 1000
0.6
0,715 2.1
511/
0.44
ACM
HIDI
13005
0.351
0.677
0.753
HK30I
0
159
0,350
3 1.113
0,777
1/5/
HIDD
0.225,
15. 130
0.361
3.423
0.61..9
0.79
3/3/0
HGDLm
0.210
13, 106
0.364:
3,375
0.624
0.801
$11/0
B111 Jas
2,857 a0
0191 a.0
1,647 -592
0.114
6150
HANO:
B.025 mn
2.519 317
0.071 mm
1,63 197
0.029 re4
0082
$11.0
So
0186 t4e
1.887 179
1,624 1.9
08822 714
0:247
196-
DBLP
HGDL
.oo
1.796 .144
0.057 .00
1,633 L.05
0,57
HGDI
0,05 ww
2
1,642= 291
0062
0.977
1330L
2,819;
OCNo.
0.342,
7. 1:000=
0.458
1.588
0 0.416_.n
066/0
HANg
0 0.451.163
5,894.1.00
0.500
3,336
0,379
5,832 1.832.0077
3/21
SaHONN
0 8004,
6.204
0.322
1.343 434
0.415
3993
06600
YELP
HCDL
m13
7.177 100
0.457
2.558 (31
0.459 m15
L.044
MGDI weake
4.342 m1
7.175 196
0.5% m16
1,557 131
111 ms
1035
WMV0
HIDL -II
0.342 .us
7.173 1220
0455 2017
1.556...31
1046 2,311
0.60
545
g
1,567 194
10700
OCNA.
8.318
..9
0.331
2/6/0
HANO:
0.497am0
9.357
0,338s
1.777
4/2/0
NOL
B 1-887
5.1%5
0:517
1.926..599
917
407/0
URBAN
HGDD
0.467
9.315
0.517
2.775
0.356
1.340
HKIDI
0 am
5.340
0.557
3.177 3,177
11955_100
1.409
4/300
HRGDL m01
0 0.5001m1
5.358 200
0.5401.010
2.776__co9
0.321_200
1.414
4/200
HGDDLa
0.48
4.325 a11
0.5274-012
2.775
01340
1,375 am7
1/50
Table 2: Mean + standard deviation results of seven models on five datasets. Best results are bold, and
t (or 4) indicates the higher (or lower) the better. Results are taken from 5 repeats. The win/tie/loss counts are suggested by the paired t-test at 90% confidence level.
8

---

CAD CH0
UM
010
CAD
0.0
100
(a) DRUG
(b) ACM
(c) DBLP
D21
IND
CC0
CAD
011
0.0
MD
HGDL
PI
Pa m
(d) YELP
(c) URBAN
Figure 3: Comparisons between HGDL results from single meta-path (CAD and CLD are calculated natural log for better visualization) for five datasets.
Loss Function KLL and Clark Distancal
Clark Distoince region,
region, upper bound
0.3
0.2
0.1
0.0
0.08
Probeaiity
Figure 4: KL and CLDD tradeoff function example. The estimated probability distribution is [x], 22. 0.9] and true probability distribution is [0.05.0.05,0.9),, with Φ + 22 = 0.1. Horizontal axis is the æ1 value and vertical axis is the loss for both CLDD and KI. divergence. Green dashed lines cover the tradeoff region where the CLD loss monotonically increases and KL-divergence decreases.
6.2 Results
Table summarzes results of all methods. Overall, HGDL wins in 99 of 180 settings, among which on average 20 out of 30 settings excel COD, CHD, IND, and KL metrics, 11 out of 30 in CAD, and 8 out of 30 in CLD. On DURG, ACM, DBLP, and URBAN datasets, HGDL outperforms its competitors in 83% settings in COD, CHD, IND, and KI metrics 46% in CAD, and 20% in CLD, Beyond its overall better comparative performance, First, HANKL and ScHGNNKL achieve better performance on YELP and DBLP dataset for CAD and CLD metrics, but on the other datasets and metries. This shows that existing meta-path based methods cannot learn distribution prediction well. In general, these models show similar performance on YELP dataset. We hypothesize that this due to the lack of rich feature information on YELP, of which the dimension of nodal features is 19 which is minimum across all datasets. Second. HGDL achieves
make the following observation on HGDL.
best results in KL-divergence by large margin across all settings. On average, HGDL have a 15% improvement compared with the second best result across all datasets in Kl.-divergence. Given that KL--dierreence the loss objective our framework, we extrapolate that HGDL converges well in terms of minimizing the distribution distance. Same observation can

---

be drawn from the validation loss curve as shown in Supplement C Figures 5, 6, and 7. In addition, on metrics being strongly related KL divergence including COD, CHD, and IND, our HGDL also enjoys significant performance improvement over other models. Among the metrics, CLD metrics shows a different patterns in terms of KI. divergence, we show in Figure $ that CL.D and KI. has a tradeoff region in small probability distribution and therefore caused such difference.
Third, the ablation study between HGDL and its variants, i.e., HGDL-srnsffomee HGDL-TH and HGDLED, demonstrate clear benefits of topology homogenization and consistency-awant graph transformer in aggregating meta-paths and nodal features for LDL learning for heterogeneous graphs (more results are deferred to the Section E. in Supplement C; there, we observe that HGDLed has no improvement in KL-divergence with different edge drop rates compared HGDL-uanstenner which is the model with O edge drop rate). We observe in Table that HGDL trunsfommer shows comparable performance on ACM and YELP by tying HGDL in five and six metrics, respectively; however, HGDL outperforms it in all settings in other three datasets. Likewise, HGDL THI ties HGDL across all metrics in YELP but is inferior to HGDL all settings in other four datasets. HGDLED ties HGDL in six and five settings on YELP and URBAN, respectively, but is outperformed by HGDL for all other three datasets in all settings. The robust performance of HGDL can be attributed to two aspects. On the one hand, the improved results over those ablation variants suggest that our devised model components for proactive meta-po patt- learning and attention modeling are indispensable. On other other hand, substantiates the usefulness of our design that lets HGDL learn semantic fusion before the embedding learning. This end-to-end learning design provides a larger search space for embedding learning to find optimal solutions, whereas other methods that learn embedding and perform fusion in two independent stages may result in suboptimal node embeddings thus inferior LDL. performance.
Fourth, even though the optimal meta-path choice may vary across different metrics and datasets, our HGDL that proactively learns to aggregate multiple meta-path graphs leads to the best performance in most cases. Figure illustrates the performance from single meta-path graph and we can observe that our method outperforms the single best path results in all five datasets, with a larger improvement when the meta-path results are close (indicating each meta-path has similar information, e.g., ACM dataset in Figure 3 (b)) and a smaller improvement when one meta- path is significantly inferior to others (e.g., DBLP dataset Figure (c) where P1 outperforms p2 with a large margin). These results validate the tightness of Theorem I by demonstrating the optimality of the learned meta-path graph in our HGDL method.
6.3 Scalability Analysis
Denote the total number of nodes, hidden dimension size, and number of meta-path by n, J. and k, respectively. The number of learnalle parameters is (n)) for graph topology homogenization, because HGDL requires learning an adjacency matrix from all meta-path, which involves knf + k2 training parameters (i.e. Onn) complexity). Inducing adjacency matrix from features, i.e. the 2nd stage, only requires (11) number of learnable parameters, same as vanilla GCN. As a result, HGDL has ((n) complexity. The runtime performance is detailed in Appendix H.3.
7 Conclusion
This paper explored a novel graph learning setting, namely, heterogeneous graph label distribution learning. Our goal is predict label distributions of target nodes in a heterogeneous graph, which enables a finer-granular delineation of node properties compared to traditional single- or multi- class node classification. We demonstrated that the topological heterogeneity and inconsistency impose unique challenge for generalizing LDL into networked data, and proposed HGDL to overcome them. Specifically, HGDL proactively aggregates meta-paths to achieve optimal graph topology homogenization through attention mechanism, followed by transformer-bant topology and feature consistency for learning node label distributions. We analyzed the PAC-Bayes ervor bound of HGDL, and the result suggests the superiority of our design over those models learned from a single meta-path graph. Empirical results on five benchmark datasets validated the tightness of our analysis and substantiate that HGDL significantly outperformed its competitors.
approach to ensure
10

---

Acknowledgment
This work has been supported in part by the National Science Foundation (NSF) under Grant Nos. IIS-2236578, IIS IIS-2236579, IIS-2302786, IIS IIS-2441449, IOS-2430224, and IOS-2446522.
References
[1] N. L. Houssou, J.-I. Guillaume, and A. Prigent, "A graph based approach for functional urban areas delineationn" in Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing.
pp. 652-658, 2019.
[2] A. Rossi, G. Barlacchi, M. Bianchini, and Lepri, "Modelling taxi drivers' behaviour for the next destination prediction," " IEEE Transactions on Intelligent Transportation Systems, vol. 21,
no. 7, pp. 2980-2989, 2019.
[3] P. Kar and P. Jain, "Suupevvised learning with similarity functions," " NewrIPS, vol. 25, 2012.
[4] A. Hefny, C. Downey, and G. J. Gordon, "Supervised learning for dynamical system learning,"
NeurIPS, vol. 28, 2015.
[5] X. Geng, "Label distribution learning," IEEE Trans. Knowl. Data Eng., vol. 28, no. 7, pp. 1734-
1748, 2016.
[6] X. Jia, W. Li, J. Liu, et al., "Label distribution learning by exploiting label correlations," in AAAI, 2018.
[7] T. Ren, X. Jia, W. Li, and S. Zhao, "Label distribution learning with label correlations via low-rank approximation,"
IJCAI, p. 3325-3331, 2019.
[8] Wang and X. Geng, "Theoretical analysis of label distribution learning," in AAAI, vol. 33, pp. 0056-52555 2019.
[9] X. Zhao, L. Qi, Y. An, and Geng, "Generalizable label distribution learning," " in Proceedings of the 31st ACM International Conference on Multimedia (MM-23), p.
2023.
[10] T. Ren, X. Jia, W. Li, L. Chen, and Z. Li, "Label distribution learning with label-specific features," Proceedings of the Twenty-Eighth International Joint Conference on Artificial In- telligence (IJCAI-19,, pp. 3318-3324, International Joint Conferences on Artificial Intelligence Organization, 2019.
[11] H. Xu. X. Liu, Q. Zhao, Y. Ma. C. Yan, and F. Dai, "Gaussian label distribution learning for spherical image object detection," in CVPR, 2023.
[12] Z. Yao, Y. Fu, B. Liu, W. Hu, and H. Xiong, "Representing urban functions through zone embedding with human mobility patterns," in IJCAI, 2018
[13] Y. Luo, F.-l. Chung, and K. Chen, "Urban region profiling via multi-graph representation learning," in CIKM, pp. 4294-4298 2022.
[14] Y. Zheng, Y. Lin, L. Zhao, T. Wu, D. Jin, and Y. Li, "Spatial planning of urban communities via deep reinforcement learning," Nature Computational Science, vol. 3, no. 9, pp. 748-762, 2023.
[15] Y. Liu, J. Ding, Y. Fu, and Y. Li, "Urbankg: An urban knowledge graph system," ACM Transactions Intelligent Systems and Technology, vol. 14, no. 4, pp. 1-25, 2023.
[16] Q. Lv, M. Ding, Q. Liu, Y. Chen, W. Feng, S. He, C. Zhou, J. Jiang, Y. Dong, and J. Tang, "Are we really making much progress? revisiting. benchmarking and refining heterogeneous graph neural networks," in SIGKDD, pp. 1150-1160, 2021.
[17] C. Zhang, D. Song, C. Huang, A. Swami, and N. Chawla, "Heterogeneous graph neural network," in Proc. of KDD, pp. 793-803, 2019.
[18] X. Fu, Zhang, Z. Meng, and King, "Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding." in WWW, pp. 2331-2341, 2020.
[19] H. Borchani, G. Varando, C. Bielza, and P. Larranaga. "A survey on multi-output regression,"
Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 5, 07 2015.
[20] Y. Wang, Y. Zhou, J. Zhu, X. Liu, W. Yan, and Z. Tian, "Contrastive label enhancement." in Proceedings of the Second International Joint Conference on Artificial Intelligence (IJCAI-23), pp. 4353-4361, 2023.

---

[21] Y. Jin, R. Gao, Y. He, and X. Zhu, "Gidl: Graph label distribution learning," in Proceedings of the 38th Annual AAAI Conference on Artificial Intelligence, 2024.
[22] Y. Dong, Z. Hu, K. Wang, Y. Sun, and Tang, "Heterogeneous network representation learning," in Proc. of the 29th International Joint Conf. on Artificial Intelligence (IJCAI-20), pp. 4861-
4867, 7 2020.
[23] Y. Jing Y. Yang, X. Wang, M. Song, and D. Tao, "Amalgamating knowledge from heterogeneous graph neural networks," in CVPR, pp. 15709-15718, 2021.
[24] X. Wang, N. Liu, H. Han, and C. Shi, "Self-supervised heterogeneous graph neural network with co-contrastive learning," in SIGKDD, pp. 1726-1736, 2021.
[25] W. Xiao, J. Houye, Chuan, W. Bai, C. Peng. Y. P., and Y. Yanfang, "Heterogeneous graph attention network." WWW, 2019.
[26] Z. Hu, Y. Dong, K. Wang. and Y. Sun, "Heterogeneous graph transformer," in Proceedings of The Web Conference, p. 2704-2710, 2020.
[27] X. Yang, M. Yan, S. Pan, X. Ye, and D. Fan, "Simple and efficient heterogeneous graph neural network," Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10816-10824, 2023.
[28] M. Wan, Y. Ouyang. L. Kaplan, and Han, "Graph regularized meta-path based transductive regression in heterogeneous information network," in SDM, pp. 918-926, 2015.
[29] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, "Graph attention networks," in ICLR, 2018.
[30] V. P. Dwivedi and X. Bresson, "A generalization of transformer networks graphs." 2021. [31] Y. Rong. W. Huang, T. Xu, and J. Huang. "Dropedge: Towards deep graph convolutional networks on node classification," International Conference on Learning Representations (ICLR), 2020.
[32] S. Kullback and R. A. Leibler, "On information and sufficiency," The annals of mathematical statistics, vol. 22, no. 1, pp. 79-86, 1951.
[33] J. Tang, J. Zhang, L.. Yao, J. Li, L. Zhang, and Z. Su, "Ametminer: Extraction and mining of academic social networks," " in KDD, pp. 990-998,770 2008
[34] Y. Gu, S. Zheng, Q. Yin, R. Jiang, and J. Li, "REDDA: Integrating multiple biological relations to heterogeneous graph neural network for drug-disease association prediction," Computers in Biology and Medicine, 11 2022.
[35] T. N. Kipf and M. Welling, "Semi-supervised classification with graph convolutional networks." in ICLR, 2017.
[36] G. Landrum, P. Tosco, B. Kelley, Ric, D. Cosgrove, sriniker, gedeck, R. Vianello, NadineSchnei- der, E. Kawashima, G. Jones, D. N, A. Dalke, B. Cole, M. Swain, S. Turk, AlexanderSave- Iyev, A. Vaucher, M. Wójeikowski, I. Take, V. F. Scalfani, D. Probst, K. Ujihara, guillaume godin, A. Pahl, R. Walker, J. Lehtivarjo, F. Berenger, jasondbiggs, and strets 123, "rdkit/rdkit: 1023_09_4 (q3 2023) release." " Jan. 2024.
[37] D. S. Himmelstein, "User-friendly extensions to mesh v1.0," Feb. 2016.
[38] V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre, "Fast unfolding of communities in large networks." Journal of statistical mechanics: theory and experiment, vol. 2008, no. 10, P10008, 2008.
[39] Q. Mao, Z. Liu, C. Liu, and J. Sun, "Hinormer: Representation learning on heterogencous information networks with graph transformer," 2023.
[40] N. Cesa-Bianchi and G. Lugosi, Prediction, learning, and games. Cambridge university press, 2006.
[41] P. I.. Bartletr, J. Foster, and M. J. Telgarsky, "Spectrally-normalize" margin bounds for neural networks," NeurIPS, vol. 30, 2017.
[42] B. Neyshabur, S. Bhojanapalli, and N. Srebro, "A pac-bayesian approach to spectrally- normalized margin bounds for neural networks," in ICLR, 2018.
[43] R. Liao, R. Urtasun, and R. Zemel, "A pac-bayesian approach to generalization bounds for graph neural networks," in ICLR, 2020.
12

---

[44] A. Tropp, -User-friendly tail bounds for sums of random matrices," Foundations of computa- tional mathematics, vol. 12. pp. 389 434, 2012.
[45] L. Bartlett and S. Mendelson, "Rademacher and gaussian complexities: Risk bounds and structural results," " Journal of Machine Learning Research, vol. 3, no. Nov, pp. 463-482, 2002.
[46] R. E. Schapire and Y. Freund, Foundations of machine leamning, pp. 23-52. Mit Press, 2012.
[47] A. Maurer, "A vector-contraction inequality for rademacher complexities," in ALT, pp. 3-17, Springer, 2016.
[48] S. M. Kakade, K. Sridharan, and A. Tewari, "On the complexity of linear prediction: Risk bounds, margin bounds, and regularization," NewrIPS, vol. 21, 2008.
[49] S.-H Cha, "Comprehensive survey on distance/similarity measures between probability density functions," City, vol. I, no. 2, p. 1, 2007.
13

---

Algorithm HGDL Algorithm
Input: G X,, Y); Meta-patha P= (Pi...., ,.....,P,. Pa
Parameter: epochs, (11, is uniform distribution, constant
Output: Y
1: A+-
2: For <each meta-path P. P>>
Meta-path Graph G, Gennerate Meta-path graph (G, P.)
4: A. + Adjacennc Matrix (G,)
5: A AUAA
6: EndFor
7: < 0.
while epochs
9:
+ Attention (A:) (Attention() follows Eq. (I))
10:
Ā Aggregate w. (., (Aggregate follows Eq. (2))
11:
Z ReLU(ÁXX;)
12:
Zg ZQ
13:
Za + (ZK)T
14:
A LapNorm (sottmax(2,,4))
15:
H + LeakyReLU(AA22..)
16:
Y + softmax (s)
17:
FAl + (Y YY)
18:
breg + (9) [1]
19:
Gradient + arg min(Exz -y. breg)
20:
Update W2, Ws, Q. K, Wi, W with Gradient.
21:
++
22:
end while
23:
return
Appendix supplemental material
B Roadmap
A structured guide navigate the supplement content is organized follows.
Supplement A: Supplement A presents the pseudo-code of proposed HGDL algorithm.
Supplement B: Supplement B reports datasets and baseline methods used for the experi- mental study.
Supplement C: Supplement C reports experimental settings and additional results.
Supplement D: Supplement D elaborates the proof of Theorem 1 as introduced in Section of the main manuscript.
Supplement Supplement elaborates the proof of Theorem as introduced Section of the main manuscript.
Supplement F: Supplement F carries out additional theoretical analysis explaining the rationality of HGDL for label distribution learning.
C Supplement A: Pseudo-code and Implementation Details
Algorithm Lists the main steps of the HGDL framework where Lines 1 to 6 generate meta-path graphs and their adjacency matrices. Lines to 10 denote the optimal graph topology homogenization (Sec 4.1 of the main manuscript), and Lines 11 to 15 are local topology and global feature consistency- aware graph transformer (Sec 4.2 the main manuscript). Lines 16 to denote the loss terms and the model training process.
14

---

Dataset
Node Types
features
Edge Types
# Labels
DRUG
DiDrug@894)SSDiieeee((55535 P:PPotei((1887yyGGcgee222220
191
DP(4397)DDS(700)
PG(118555/PPP200133)
GG17122566/DDD00983316
ACM
AAAtthok($1100PPPPpcc((219999999
1903
APC37055)/CC122499
14
5.Suhjeu(77//Ccooffreeeee(())
PS(12499)/F((12499)
F.Affiliation((100)
AF(30424)
DBLP
A: A
8920
APC19645/PT((14273)
U: TTTemm(8920//cconffee
PC(14328)
YELP
UR(3001 (/RB(6990228))
R: RRReriee.6990220177TT00000001
UT(3001 (BB((908115)
URBAN
NR:Nature Residence(622)
155
NR-CS (14818)
CS: CS:Comprehensive Service(449)
NR-GL (8328)
GL:Green Leisure(202)
NR-TJ (11736)
TJ:Transit Junction(161)
Table Benchmark dataset data statistics
Datasets
Meta-Paths
Label semantics
DRUG
dpd, dsd, dpgpd apa, afa, apspa
See Table 6
ACM
(KDD, SIGMOD, WWW, SIGIR, CIKM, SODA, STOC, SOSP, SPAA, SIGCOMM, MobiCOMM, ICML, COLT,
VLDB]
DBLP
apepa, aptpa urbru, utbtu rsr, rir, rtr
{ (Database, Data Mining, AI, Information Retrieval)
YELP URBAN
{1, 1.5, 2, 2.5, 3, 3.5, 4, 5)
[ |Accommodation and Residence, Workplace and Public Service, Sustenance and Leisure Places, Farmland, Trans- portation, Business and Industry, Education, Hcalthcare, Greenfield, Worship)
Table 4: Meta-paths and label semantics of the benchmark datasets
D Supplement B: Experiments and Datasets
D.I Dataset Description
Because no benchmark heterogeneous graph datasets with label distributions currently exist, create four datasets using existing heterogeneous graphs including DBLP [33], ACM (33] YELP, and DRUG dataset 34 Table reports number of nodes/edges, number of each nodeledge types, and number of labels for each dataset. Table reports meta-paths and label semantic for each datasets. Number of features are refer the content attribute/fcatures of the target node types. In our experiments, we did not use any content information of non-target nodes (so only target nodes have features).
DBLP is citation network 33 with four types of nodes (author, paper, conference, and term nodes). Author nodes are our target nodes. To construct a label distribution, we labeled each author with a distribution of conference areas, using the history of each author's published papers at different conference areas. The author's node features are bag of words of all papers published by the author.
ACM a citation network from [3]] with five types of nodes (author, paper, conference, affiliation, and subject nodes). Author nodes of the authors and then labeled them based on the subjects of their published papers. Similar the DBLP dataset, each author's features also bag-of-word representation of the author's papers.
target nodes. We first extract one third
YELP an open-access review network with four types nodes (business, user, tip, and review nodes). User nodes are our target nodes. We label each user by counting the ratings of the business each user made reviews on and normalize the countings of each rate to distribution. User's s features are the bag of words of their reviews.
15

---

Drug name
Drug ID
Drug SMILES
Clobazam
DB00734
Vinorelbine
DB00987
Carvedilol DB00812
Table 5: Examples of drug name, ID, and drug in SMILES notation (SMILES: Simplified Molecular
Input Line Entry System)
DRUG prefix McSH code
Name
C01
bacterial infections and mycoses
C02
virus diseases
C03
parasitic diseases
C04
neoplasms
C05
musculoskeletal discases digestive system discases stomatognathic diseases
C06
C07
C08
respiratory tract diseases otorhinolaryngologi diseases nervous system diseases
C09
C10
C11
eye diseases
C12
urologic and male genital diseases
C13
female genital diseases and pregnancy complications
C14
cardiovascular diseases
C15
hemic and lymphatic discases congenital, hereditary, and neonatal diseases and abnormalities
C16
C17
skin and connective tissue diseases nutritional and metabolic diseases
C18
C19
endocrine system diseases
C20
immune system diseases
C22
animal diseases
C23
pathological conditions, signs and symptoms Chemically-Inducee Disorders
C25
C26
Wounds and Injuries
E01
Diagnosis
F01
Behavior and Behavior Mechanisms
F03
Mental Disorders
G07
Physiological Phenomena
Table 6: DRUG label semantics
Class
Class 2
Class
Class 4
0.8774 + 0.1755
0.0415 + 0.1086
0.0131 + 0.0619
0.0680 + 0.1226
0,0628 + 0.1188
0.7733 + 0.2157
0.0767 + 0.1463
0.0873 + 0.1536
0.0161 0.0576
0.0438 + 0.1023
0.8667 = 1811
0.0734 + 0.1472
0.0309 = 0.0879
0.0364 + 0.0907
0.0266 = 0.0757
0.9061 = 0.1622
Table 7: DBLP dataset average label distributions (tt) for all nodes w.r.. different classes.
To generate this average, label distributions are grouped by their dominant class. Average label distributions are then calculated within their respective groups. Standard deviations are calculated between each class within each individual group. The table is q q. where q denotes the number of classes. The diagonal values denote the dominant class's 's probability value. The lower the diagonal values, the more spread out the class probability is.
DRUG a saaug-disaaseproteinn -een network from (34), with four types of nodes (drug, disease, protein, and gene nodes). Drug nodes are our target nodes. The label distribution each drug node is obtained by using associated disease etiology (using top-level cate- gories from Medical Subject Headings, McSH, disease database). The node features is
16

---

Olr
1007 115
1011 -0300
03399 89584
0003 10377
60005:00 0.904
1009
0001 0018
385
10134
0800
3064
000
1096
0001
0.004
0001
1037
100
1300 -114
1.00
13m -010
0888
1100
0100
0309
1000-1
011
-121
103
1000
1104
2440 3334
1101 -1010
834 41000
0885
012
3130
01000
0312 011004
21300 1:844
1000-188
21.00
18:00
1126
2000 0001-1-004
0.800 -1306
Table 8: ACM dataset average label distributions Std) for all nodes w.r.I. different classes. To generate this average, label distributions are grouped by their dominant class. Average label distributions are then calculated within their respective groups. Standard deviations are calculated between each class within each individual group. The table is q x g, where y denotes the number of classes. The diagonal values denote the dominant class's s probability value. The lower the diagonal values, the more spread out the class probability is. Class 7 has 0 label distribution because samples have Class 7 as the dominate class.
Dlas:
Clard
Clac.
Clar
Car12
21811
10:00
84272
04533
3119
0360
4:45
01015
1/0.9
8.005 8646
E002 3414
32390
L0W5
0418 26xTs
0311
60811 1471
E/211 8X15
84330
20315
04583
16.319
0348
0100
1.44
8.016 8190
1001
540% 1.300
10-25
0419-000
0134
60015
144
010111100 9415
14813
08273
11211
2112
1120 48-42
61872 0101-3
61881
0410
11191
01307
6-0744000
21991
14012
03213
11000
0.003
51124
E101
5106
0428
5.0014
1151 218LL
03764
63794 01114
E001 307%
100119
1179
0010
0388
E3011 2111
1941
1180
1378
4085
0208
0330
E3624
3007
0838 2064
L128
Table 9: DRUG dataset average label distributtionspaart 1) (mean=:Std) for all nodes w.s.l different classes. To generate this average, label distributions are grouped by their dominant class. Average la- bel distributions are then calculated within their respective groups. Standard deviations are calculated between each class within each individual group. The table is q x q, where q denotes the number of classes. The diagonal values denote the dominant class's 's probability value. The lower the diagonal values, the more spread out the class probability is.
01111
5.1155
0U251
8-3177-
2312
1.821
55155
1.86
00015
02.00
1.333 4.
0/030 0.900
01381 8.000
9
1.815
1.6
040-0-
83111
8.3103
83184
00570
88221
10334
53355
00010
0130
5.536
03311 115100
83112
01011
2.741
0001 0130
0255 6.58)
9
59155 57117
00051
8.114--801 1181
01000
0109111000
0155 010
03135-44 83159
3.31
10000
196
0230
10064
8355
10215
0136
1840 2350
01911
3
0107 075-5
01303 0.000
0
1.96
10T7 00015
2505
WOTHE 11.0
E807
03133
(1262-
19211
3.8%
10003 00-.3
02000
02.57
1355
LANE J 2.125
(U231 00054
03115 631.3
15221
2006
(UD0 00C53
28305
00015 0236
18336
0103 00009
0131 00000
0
1011 33111
01811 1-00
03117 1111
11:11
100T 00:43
1966
1080 1100
01311 01111
63122
3806
2009 2411
0101 6M0M
0-135
80051
0600
00-33
0/8 1100-
01315 88111
4 50115
OU051 L 00064
8-115 $21-0
(UCH -UO
3.221
2
(UD12 6UC53
2344 20196
00019 OUDIC
0.830 2
00030 00099
0 00007 0
Table 10: DRUG dataset average label distributions(part 2) (tt) for all nodes w.r.f different classes. To generate this average, label distributions are grouped by their dominant class. Average label distributions are then calculated within their respective groups. Standard deviations are cal- culated between each class within each individual group. The table is q x q, where q denotes the number of classes. The diagonal values denote the dominant class's 's probability value. The lower the diagonal values, the more spread out the class probability is. Class 21, Class 25, and Class 28 has label distribution because no samples have any one of these three three classes (21, 25, 28) as the dominate class
created from chemical compound of each drug using their SMILES notations. SMILES (Simplified Molecular Input Line Entry System) is a chemical notation that represents a chemical strueture as string it can be used by the computer. Table lists examples
17

---

Clux
Class
Class 5
Class
Class 5
6
Class
Clec
Close
0,616 1: 0.2722
0.522 = 0.0852
0.0097
0,940
I11056 = 0.1234
0,0255 2: 0.0774
0,0099 2: 3000521
0.0000 = (10000
0,1324
0,0855
1,0041 4: 0.0770
0,0938
1 0.1400
0.6153 = 0.28211
0.019 a 0.0634
1.1046 2 -0.3170
0.0117 2000485
0,000 2: -0,0000
0,0000 = -1.000
0,1595
0.0975
0.0144
4: 1000315
0,090 0,0848
0.0409 = 0.0924
0.7113 a 0.2097
0U2255
4 +0.0490
0,0021 1-0.1899
0,009
-00052)
0.0000 = 0.0000
0.0368
0.071
0,0074
1:0.020
0,0675 1 0.1799
0.1585 = 0.1725
0,1130 001811
06573 1 0.26638
0.0212 1: 000016
0:221
-0.0720
0,000 = 0.0000
0,0695
1 0.1100
0.0144
1 10.0815
0:057 0:0816
00198 = 0.0755
0,0117
0.0699
00196
1 0.0633
0.8670 1: 0.2524
0:000
0.0000
0,000
=: 0.0000
0,0054
0.0285
0,002
1 0.0067
0,053 a 0.0358
0.085
= 0.6499
0,0122
0.085
0.071
2 0.0488
0.0076 5 00057
0.521
2 6022455
0,000
= 0.0000
0,0099
0.0022
0.022
1 0.0167
0:011 0.0199
00017
= 0,288
0.0014
0.0168
0.014
+ 0.0221
0,0000
+ -000000
0:011
0000000 0.1044
08750
0.2315
0.0025
0.0169
0.000
+ 0.0000
00481 + 0.3050
00830
0.100
0.0686
0.1510
0.6771
+ 1010
0.0446
+ 1255
0.052
0..122
00825
0.1 768
0,7568
0.2572
0.0090
+ 0.0417
0.0188 + 0.0023
0.186-
= 0.0631
0.0475
0.1352
0.019 + 0.0455
0.0203 + 0.0970
0.0455
+ 003444
0.0255
= 0.1768
0.0343 + 0.0022
0.9273 + 0.1952
Table 11: YELP dataset average label distributions (td) for all nodes w.r.l. different classes.
To generate this average, label distributions are grouped by their dominant class. Average label distributions are then calculated within their respective groups. Standard deviations are calculated between each class within each individual group. The table is q x q, where q denotes the number of classes. The diagonal values denote the dominant class's 's probability value. The lower the diagonal values, the more spread out the class probability is. 0 indicates no samples have specific dominate class.
Claw
Class 2
Clac:1
Clos
Clacs
Clae
Clos?
Casch
Claw
Clos 10
0.0855 4.93113
0.0300
019155
0,0023 4- 6-007797
0.0750 00000--0100 0:131
B00ST - 0.0011
4.0.00
00038g
0:033 - -0.044
01071
4-0.316
0:0000
0.0044 041187
0.8157
11.3111
0.3942 +: 0.1945
0,0294 = 0.2452
0J6222 = 0,1020
0.6621 2.0.100
0,388 = 0.0742
[L8238 -0.003
0.0471 C.025556
0,0244 0.845
0.0650 2 0.0880
000350 = 1181220
RU6177 2: 0.1745
0.046620..000
EUM:2 = 101255
0.0001 2-0.254
0 00.875
0U0029 2 20,422
3.100
2..144)
0 00945
0000=
D.GI15
0:8724
0.0965
c-00ma
0,485 2.
0.3665 000664
0.310)
2.0,00
0.0037 =
0.0001
0.12%6 -.0655
0.0857
#:1144
0.226 0001s
033807 0.1551
0.0151
= 0.075T
0.0123:::000 0.0671
0,0009 = 0.0107
= 0.1252
010012 2-23122
00:50:000551
0.005 = 0.0225
3.586L 2:1.30
0.0000 00030
0.081 9.322
000:20
018522
DJsSSS J 00.116
B0075
L 0.1150
00087 1 C.0066
0/5:71
0.881
0.1033
0.022
0.0S5 01056
0:008:1 1.03373
0,9022
0.0455
003812
0.1965 0.1355
00061 1-00000
DA332 0.075
0.658
0.2175
06088 1.00000
01001
01112
0.048 0.686
0.0166 0.895
(16225 0L0061
0.3627 0.135
0.0.8 0.005
04000 1.660m
0.0000 2 00.255
0.00.9 2: 0.003
06671 1.00000
0.:00 0.199
Table 12: URBAN dataset average label distributions (tea for all nodes W.E.f. different classes.
To generate this average, label distributions are grouped by their dominant class. Average label
distributions are then calculated within their respective groups. Standard deviations calculated between each class within each individual group. The table is q x q, where denotes the number of classes. The diagonal values denote the dominant class's probability value. The lower the diagonal values, the more spread out the class probability is. O indicates no samples have specific dominate class.
of three drugs' names, IDs, and their SMILES strings. Because SMILES describes drug's s chemical compound, We use RDKit library (36) to convert the chemical compound to 191 measures(scalars) of that chemical compound and use it as drug node (target node) features. To obtain the label, we use the McSH-tree parsing tool (37) to convert the disease ID to corresponding MeSH prefix code which corresponds high level disease and obtain in total 28 high-level diseases that drugs could potentially target on. Table lists the 28 labels used in DRUG dataset (including MeSH prefix codes and names).
URBAN is functional urban area network constructed based on the city of Porto, Portugal, uniformly divided into a grid of 30 by 80 blocks. A dataset comprising 22,208 points of interest (POI) is sourced from the OpenStreetMap platform P to generate the label distribution. This dataset includes a large amount of fundamental categories like hotels, banks and churches, which have been further categorized into 10 more general groups, as detailed in Table 4 We count the types and corresponding numbers of POIs to generate the label for each block. After excluding blocks devoid of any POIs, the remaining blocks were primarily divided into four different types of nodes using the Louvain algorithm (38). These node types are named as Nature Residence, Comprehensive Service Green Leisure and Transit Junction based on their main functions and nodes from Nature Residence are our target nodes. The edges are constructed by Taxi Service Trajectory P which documents the taxi routines (from origins to destinations) in Porto. In our resulting heterogeneous graph, two target nodes are considered neighbors if they have the same recorded destination.
Tables 7
and 10 and I1 report average label distributions for DBLP, ACM. DRUG, YELP, and URBAN dataset, respectively. To generate average label distributions, for each dataset, nodes are grouped by their dominant class (where dominant class the class with the largest distribution value among all classes, using ground-truth distributions). Average label distributions are then calculated within respective groups. It is worth noting that some classes may not be a domain class of any
www. www.ooeenttreemapp.oo
https:/ h/www.///www../ kaggle. wtt.c//c//ccccecc wttp////pppddd--ppddttttt---- predictreect---e seattttaxiirrevvee http://ccettaaactor---//rs-e i/data
18

---

node (e.g. Class 7 of the ACM dataset in Table 8 or Classes 21, 25, and 28 in the DRUG dataset in
Table I0.. In this case, the average label distributions of the class are reported in the tables.
D.2 Baselines
We extend existing heterogeneous graph learning methods to integrate KL-divergence loss, and implement three baselines for comparisons. In addition, three variants of the proposed HGDL method are compared for ablation study purposes:
GCNKL: A baseline uses graph constructed from each meta-path to train vanilla GCN, using KL-diveegence as loss function, and reports the best meta-path result. This baseline is used to demonstrate whether the proposed HGDL can outperform the best single meta-path result.
HANKL: This baseline is to compare HGDL's 's semantic fusion component with HAN [25] a classical heterogeneous network learning semantic fusion method. HANKı. uses HAN to integrate embedding from meta-path graphs created from different meta-paths, and uses KL-divergence the loss function for training.
SeHGNNK1. This baseline is to compare HGDL's 's semantic fusion with SeHGNN 27 a state of the art transformer based heterogeneous semantic fusion learning method. ScHGNNKL uses neighbor aggregation proposed by SeHGNN to aggregate meta- meta-paths embedding, with KL-divergence being used as the loss function for training.
HGDL: The proposed HGDL method.
HGDL-TH: This is a variant of the HGDL for ablation study. HGDL-IH removes the graph topology homogenization module (Sec 4.1). The embedding trained from each meta-pal path with topology and feature consistence-aware graph transformer (Sec 4.2) being applied to each meta-path. The result of the best meta-path reported, and the purpose to demonstrate HGDL's topology homogenization effectiveness.
HGDL dransfommer This is a variant of the HGDL for ablation study. HGDL dransformer uses GCN, instead of the transformer (Sec 4.2), to learn embeddings. This serves as ablation study to show the advantage of HGDL's transformer component for embedding learning.
HGDLLED This
variant of the HGDL for ablation study. HGDL-ED replaces HGDL's topology and feature consistence-aware graph transformer (Sec. 4.2) using random edge dropout method (31). The purposes to demonstrate the advantage of consistent-aware graph transfer, which resembles to learnable edge dropping, compared to random edge dropout.
D.3 Experiment Setup
All results reported in the paper are based on 5 times average for each method. Hyperparameters of all the baselines include the hidden dimension for the embedding learning and the hidden dimension for the semantic fusion learning (if the semantic fusion part exists), and the learning rate. In practice, a 0.005 learning rate is fixed for all the baselines unless it is observed that specific method cannot converge under such learning rate or can converge much faster with a large learning rate. An early stop patience signal is set and maximum number of epochs is fixed. In practice, we observed convergence before reaching maximum number of epochs in general. For the methods that cannot learn well with all the hyperparameter search, we report the best results we got. For the HDGLED method, the originally learned feature topology for HGDL is replaced by 0-1 matrix generated by Bernoulli distribution with drop rate provided to convert to a 0-1 matrix. The drop rate for the HDGL baseline in Table 2 in the main manuscript is 0.1. Fig 3 provides the results of different drop rate cases. For HGDL we have additional hyperparameters negative slope of Leaky Relu activation and y for regularization loss.
The train:validatton.tess splits for DRUG, ACM, DBLP, YELP, and URBAN dataset are 8:1:1, 7:1:2, 4:1:5, 8:1:1, 5:3:2, respectively. We used a smaller training set (40%) for DBLP because because this dataset has comprehensive nodal features (8,920 features) which are the-bag-or oo-ds of papers published by authors. DBLP also has much less number of labels (4 classes), compared to other datasets. The rich nodal features, combined with less number of classes, provide rather rich information to derive research fields. So we reduce the training set size, compared to the test set, to provide less label information for learning.
19

---

D.4 Evaluation metric
Given one predicted label distribution ÿ and its corresponding ground truth distribution y, the six metrics used in the Table 2 can be defined as the following:
Cosine Distance(COD):
COD(y. =1
(4)
11
Canberra Distance(CAD):
1ife Di
CAD(y, 0)
Σ
(5)
M/ + 10A
Chebyshev Distance(CHD):
CHD 0) max 1/A Dz
(6)
Clark Distance (CLD):
CLD(Y, 0) (yi 0z)²
(7)
(3 + ūz )
Intersection Score(IND):
IND 0)
Σ min U1)
(8)
Kullback-Leibler Divergence(KL))
y1 kl 9) = Σ y log
(9)
E
Supplement C: Additional Results and Analysis
E.1
Edge Dropping Comparisons
HGDL treats the learnt feature attention as another adjacency matrix, and incorporates it into the graph adjacency matrix by @ operator, which provides an intersection of neighbor sets for node, i.e., the resulting convolution layer is propagated to a neighbor set that both feature and graph topology agrees on. This operation is similar to the recently proposed drop message idea that randomly masks edges for the nodes. Instead of random masking, HGDL selects the neighbors that both feature and graph topology agree on. In other words, Eq. in the main manuscript functions similarly to the edge dropout [31]; in lieu of randomly removing edges, we enforce neighbor-set intersection, where the information is only propagated from the neighbors which the feature space and meta-path topology both agree.
To compare the performance between our approach vs. the edge dropout 31 the variant HGDLED used replace our transformer based approach by using random edge dropout. this case, for HDGLED the originally learned feature topology for HGDL is replaced by a 0-1 matrix generated by a Bernoulli distribution with a drop rate provided to convert to a 0-1 matrix. For comparisions, we vary the edge drop rates and report HGDLED'S 's performance in Figure 5 The results show that that HGDLED with random edge dropping doesn't necessarily help improve the distribution learning, implying that our method works not just by relying the decreasing of the edge number, but also relying on choosing the proper edges to drop.
E.2 Validation Loss Comparisons
Figures 6 and & report the training process validation loss (w.E. KL-divergence loss) between HANKI. SeHGNNKL. and HGDL. Because early stop is applied to all methods, they terminate at respective number of epochs. In our experiments, patience score is used to determine the minimum loss. method will continue if a smaller validation loss found within patience number of epochs after the current smallest loss, and terminates otherwise.
20

---

DRUG
Model
COD4
CAD4
CHD
CLD4
INDI
KL,
GCNKL
0.220
9.209
0.245
1.9634.131
0.676 029
0.4844.073
HANKL
0.279+.023
10.084 823
0.268+.027
2.155
0.632 025
0.579+.053
SeHGNNKL.
0.286
10. 2811
0.267 2.020
2.166.1.13
0.640
0.600
GLDL
0.199
8.887+.838 +.877.888
0.232 .014
1.89 =.17
0.699 019
0.409
HINormer
0.3651.026
10.5631.571
0.308 1.029
2.227 1.009
0.5751..02
0.7421.
HGDL
0.168#.019
9.1794.574
0.217
1.957+.114
0.710 020
0.392 0414
HGDL transformer
0.199
9.371 1.679
0.235 1.017
2.004
0.687 166871.021
0.4921.050
HGDL-1
0.212 023
9.510+.00
0.240
2.029
0.67 + 020
0.462 .000
HGDL
0.204
9.602 1.882
0.239 1.028
2.040
0.681 .000
0.574
DBLP
Model
COD
CAD4
CHD
CLD1
INDI
KL,|.
GCNKL
0.031
2.8524.000
0.091
1.647+.002
0,908+.006
0.1 0.114
HANKL
0.0254.002
2.8194.012
0.071
1.6334.007
0.929#.004
0.082
ScHGNNKL
0.086+.140
2.887
0.155
1.6244.049
0.8424.214
0.252+.307
GLDL
0.019
2.84.009
0.056#.002
1.6334..01
0.943 0003
0.06+. .003
HINormer
0.053
2.9364.013
0. 0.1464.019
1.6694.00s
0.853 .019
0.22
HGDL
0.019
2.7964.014
0.057 2.05
1.6334.00
0.943 005
0.057+.011
HGDL Atransformer
0.025+.000
2.828 1.04
0.0744.005
1.6424.001
0.925
0.090
HGDL-E HGDLad vdd
0.020
2.808+.013
0.062+.00s
1.637
0.937 005
0.070
0.0234.001
2.819 2.8194.005
0.0704.00x
1.6394.003
0.929
0.0821.006
Table 13: Experimental results of HINormer and GLDL the Drug and DBLP datasets, where
HINormer a recent heterogeneous graph learning baseline and GLDL is for LDL homogeneous graphs
DRUG
10 loes
0.4
0.6
0.5
edge drop rabe
Figure 5: HGDLED baseline with different edge drop rates DRUG and ACM dataset.
21

---

HBDL
SeHGGNNe
0.-
2.6
420
300
000
770
Epochs
Figure 6: Training process KL-divergence validation loss comparisons the DRUG dataset. The --axis denotes epochs, and the y-axis denotes training validation loss. Early stop is applied to all three methods, methods terminate different epochs. A method terminates at the minimum loss, no smaller loss found after continuing patience number epochs.
HANo:
HGDL
SeHGGNN
0.6
9.-
Epochs
Figure 7: Training process KL-divergence validation loss comparisons on the DBLP dataset. The rraxis denotes epochs, and the y-axis denotes training validation loss. Early stop is applied to all three methods, so methods terminate at different epochs. A method terminates at the minimum loss, no smaller loss found after continuing patience number of epochs.
HANeL HGDL
SeHGNNKe
521 Valddetion lass
Epochs
Figure 8: Training process KL.-divergence validation loss comparisons on the ACM dataset. The rraxxi denotes epochs, and the y-axis denotes training validation loss. Early stop is applied to all three methods, so methods terminate at different epochs. A method terminates at the minimum loss, if no smaller loss is found after continuing patience number of epochs.
22

---

utban
1.5
K.Divergence
1.0
0.0
-0
-3
logly)
Figure 9: Sensitive analysis of 7, ranging from 0, le-5, le-4, 1e-3, le-2, and 0.1, converted the values of log(7) as -6, -5, -4, -3, -2, and -1 along the X-axis.
The comparisons show that HGDL achieves the smallest Kl.-divergence loss across all datasets. For the DRUG dataset (Figure 6). HGDL converges much better optimal point than the other two baselines. This is also evident by the results shown in the Table 2 of the main manuscript (which are validated on the nodes).
E.3 Sensitive Analysis of hyperparameter
To study the effect of hyperparameter ? on the model performance, we explore the y value ranging from (0.le-5,1e-4,00 1c-4.le-3.1e-2,0.11 and see how the KL divergence metric is affected by different values for our five datasets shown in Figure We can observe that optimal y values vary for different datasets and therefore it is important to search for proper value for each datasets.
E.4 Comparison against HINormer and GLDL
HINormer 39) is recent heterogenous graph learning baseline and GLDL method recent LDL learning method in homogeneous graphs. To make a more comprehensive comparisons, Table [3] provides the results applied DBLP and Drug dataset with these two additional baselines. We observe that our method is still competitive in most metrics in terms of GLDL and is more superior HoNomeer method in general. This further validates the superiority of our design on heterogenous settings.
Supplement D: Proof of Theorem
Theorem 3. Let (E(L(x.A.) (h) be the empirical risk of using the i-th meta-path graph A; Afo label distribution prediction. With the SGD step-size 7) we have
In k
((x,))(t) I min A.AA
((L(
+
Proof. We define the cumulative risk L(x.A,) (fi
En (1)]) over n training iterations, and a quantitative variable Qn as follows:
On =xp((mnLxx.,,(t)) +... exp(-nL(x.A.)
(10)
where it is trivial to derive Q: = ki as no risk will be suffered if no target node attends training. By performing stochastic gradient descent (SGD) over n iterations, we have
In Qn In Σ exp(-nL4xx.,) - In Q
A.EA
2 In max { exp( (t)))))) In A.EA
1 min (L(x.A)(1)} Ink.
(11)
ArEA
23

---

and
Qn
EAEA exp( (77 (x,A, (h) (, Un]]]
(2-1)
In
In
Qn-1
EA,Ed exp(-1) (n-1)
(x,A,) (fi)
In
softmax (1nimax(-n/ (n-1)
(X,A:) (fi) 77: Mn))
A,EA
Σ (g..0eep(-nnp1p Un)), A.EA
(12)
where (9h,, Mn)) E(K(X, [r], yn.) denotes the immediate loss evaluated the n-th training iteration based on the i-th meta-path graph topology. The attention matrix θ are with all entries non-negative by its definition. To proceed, we introduce the lemma by 44
Lemma 4 (Hoeffding Inequality). Let X be a random variable with a S X S Then for any E R
In S SEX (b)
8
With normalized immediate loss (8n, Un) E [0, 1], Vi, we can relax Eq. of l and h:
based on the convexity
Qn S -nl Σ
n²
In
O1; Bmn)
Qn-1 n-1
8
A:EA
k
S _n Σ [1.:) [n] + 7
S [e( vn)]
(13)
We can observe that over training iterations:
On
en-1
Q2
In
+ In
+... + In
Qn-1
Qn-2
Q1
Qn Qn-1
Q2
Qn
in
= In
Qn-1
Qn-2
Q
Q1
n 7
S ((h(X,
(14)
8
Chaining Eq. and Eq. [4] we arrive at:
((((X s min (1))) + In + A.A
n
8
and dividing the both sides by n, we have
In l
(k) & min (L)
+
A.EA
Note that E[L(x.A, ((4.).)) concludes the proof.
G Supplement E: Proof of Theorem 2
Theorem Let h H - IR. an 1-layer message-passing neural network with maximum hidden dimension k of which the i-th layer is parameterized by W., Then for any 8, 7 B > 0 and
24

---

>> 1, with probability at least - 8 we have
2(v2q + v2)g
Lgg (l))
-I((x,) (h) S
max (E(n]JE(
h
Vn.
1/6
+ +0
+0(1
B2d B2 112k (I) log m2
36y
2n
7
where D(W,) =
[ i=1 [Will2
El
(II (
bounds the hypothesis space and b is [W::1
constant.
Proof We first try to explore the boundary of GCN without label distribution, and use the multi-
class y-margin loss following 41 42]. The generalization error and the empirical error are defined as,
Lg (fw) = P InD
fw (X, A)lvl & + max fu (X, A)6l) s7y
(15)
((X,A) (Fw)) = Σ #LES
[x,( S max fw A)lil 144
(X,A))|]
where > 0 and (x, A) is the I-th layer representations, i.c. Hi fo (X, A). 1 is all-one vector. Two necessary lemmas are derived from this. First, a margin-based generalization bound is given to guarantee that, as long as the change of the output brought by the perturbations is small with a large probability, the corresponding generalization bound 42 defined as:
Lemma 6. Let (2): X R be any model with parameters w, and let P be any distribution on the parameters that is independent of the training data. For any w, we construct a posterior Q(w + +u)
by adding any random perturbation u to w. s.I., P (max (Jwt(()) - fx(()) x < ()) > 2- Then,
for any y, 5 O, with probability least 1 8 we have:
2Kk (Ql + u)]lP) log 8
Lg (fw) S ((x,A) (fw) +
(16)
2(n - 1)
where is the number of instances of training set. Hence, in order to apply Lemma 6. we must ensure that the change of the output brought by the weight perturbations is small with a large probability. In the following lemma, we bound this change using the product of the spectral norms of learned weights at each layer and a term depending on some statistics of the graph [43)
Lemma 7. For any B > 0,1 > 1, les f E H X x - R be I-layer GCN. Then for any w, and E XB hos and any perturbation the output of GCN is bounded as,
vec (1U)))=1) such that Vi E No ||Uilla $ [|Wall the change in
((x.A) (Flotui) L(X.A) (fw)l2
S seBa'd
I
|1
(17)
[Wall2
Then, we can deduce the bound for GCN without label distribution. First, let B (IT' (Wall)) 1/t. Weights are normalized as W. T ReLU, i.e., az(x) = ø(az), Va 2 0. IT ||Wil IT' [ill_ and [Wills/ ||W5ll
8 W.. Due to the homogeneity of
W: wall W: i.e., the terms present the bound remain unchanged after normalization, There- fore, w.l.o.g., we conduct the assumption that the norm equal across all layers, i.e., Vi, ||Will, 8. Consider the prior = N (0, o²1) and the random perturbation u ~ V (0, o² Note that the a of the prior and the perturbation keep the same and will be set according to 3. Specifically, the value of a is set based on some approximation 3 of 3 since the prior P can not be directly deduced by any learned weights. We select the approximation 3 as the cover set that encompasses the possible significant range of 3. For the moment, assuming we have constant 3, we can examine that
25

---

fulfills the condition |3 B| S 1.p. Note that this also implies
1
B S B $ (1+ 8,
| - - 7 3 =
1 (1-7
1
1
1-1
8 S 31-- s (1 | 7 B
1 1-1
1
1
(18)
1 1
8 S 31-1 v (1 /
1
8
B v 81-1 S eg-
According to 44 for random perturbations Ug E hh which is object to the distribution U, ~ N (0, PP (1UU, 2 1) S 2ke 8 //2k²
(19)
We consider the perturbations of all layers, thus:
P (1111122 < t& & l|Uill2 11)11PPP2 |1 2 t),
2 1 Σ P (1|Uill2 2 t)
(20)
2 21ke -t? - o2k
Setting 21ke -f /2ko²² J/2 can have t / log(4l))).
The likelihood that the spectral norm perturbation any layer not exceeds / 2h log(41k) is least }. Plugging this bound into Lemma 7
we have with probability least 1
(lw+a(X. A) fw fx A)l2 S =
W
||0|||
eBd 11
g
3
(21)
i=1
< eBd 1-1
301-1"1- lo / 2k log (4k),
< è² Bdd 31-1 / 2k (1kk,,
? 4'
where we can set o 4e Bd 1 B1 11 log(4[k)
to get new inequality. Note that Lemma 7 also
requires Vi Nr, 1 ||W The requirement is satisfied if can be satisfied if
which in turn
1/ log(44t))
<
(22)
4e Bed - 31- 11 2k (4ik)
2k log 4lk)
since the chosen value satisfies S
Be - gt 11 2k log (41k)
26

---

Note that Eq. (22 can be rewritten deduced as:
d
I B¹. KL term in the PAC-Bayes bound in Lemma
1eB
(T)QQ)))
W
2
4e B2d 3-20011---222 log (41k)
Wall
2x²
2-²
Bdd g2 log(lk)
||
0
(23)
72
12kd-112klog(1
II 11Wil12
Σ
72
|will
From Lemma 6 fixing any B, with probability 1 - å and for all such that |3 -
- s 18.. Let
D(W,) = II willl E/= / (IWall2) we can get,
Lg (Jw) S ((x.)) (fw)
u2k log()))((W,) log 8
(24)
+o
22n
The choice of 3 is important, such that for any B, can bound the generalization enror like Eq. First, the value range of 3 is decided as the following,
24
1/1
1/1
rvd 2B
7/ nd
SBS Vd
(25)
vd
2B
since otherwise the bound holds trivially as ((x,4) (fw) S by definition. Note that the lower bound in Eq. (25) ensures that Eq. (22) holds which in turn justifies the applicability of Lemma If
I/1
v
then for any (x, A) and any j N |f(X, A)Dil s z. To prove this, have,
Vd
2B
(X, A)DI S f (X, A) = 1,," Wi
2
1 11nx-111 1/ ||Will2.
n
< [Willz max |x- -i, 1)
(26)
i
Bd
[Wll2 = - B²B,
d 1-1 B
Y ?
2Bd 2
TVd 1/1 2B
Therefore, always have L(x.A) (fw) when B
<
Alternatively, if 3
Vd
1/1
2vnd 28
the term inside the big-O notation in Eq. 24
would be.
vd
B2d ( + log n
1'k Pllogg())
Wall
72n
4
[
(27)
12k log(lk)
2 1,
where we set k 2 2 in practice and 2 2, while considering the fact ||W F 2 |W 2- To account
for within the range defined by Eq. 25 condition ensuring that | 3 < 7 would be
27

---

1/1
18
Ivá 2B
So, if we can identify an overlap of the span in Eq. (25 with radius of
Iyd 1/1
0v7 2B
and ascertain that bounds akin to Eq. (24) met when 3 assumes any value within
I/d
that overlap, then a bound that is valid for every / can be established. Clearly, it is only essential
to ponder coverage C' of magnitude |C] = 2 n -1 1). Thus, representing the event of Eq. (24
when 3 adopts the i-th value from the coverage E,, we deduce:
P (E &E)) = (i, Es)
|0
(28)
2 1 P E 2
1 1
Note E; denotes the complement of E.. Hence, now present the PAC-Bayes generalization
bound [43] of GCNs as Theorem [
Theorem 8. For any B > 0, let fw E H. X + BK be I layer GCN. Then for any 5, y > with probability at least ó over the choice of an i...d size-n training set according to
D. for any w. have,
Lg (fw) S L(x,A) (fw)
C Bad B2d 182k log( + log 5
+
2Pn
(29)
= L(x,A) (fw)
B2d log()) log ml & (1)
2n
Then we consider a GCN with label distribution learning, and explore the bound in term of Rademacher complexity. Specifically, let 17(-) be a label distribution function, thus 7) (x;) {uk).... !x }- Given a function class H and a loss function E(-), for function h E H. its corre- sponding risk and empirical risk are defined as Lg (fw) = Ex~D[e(f((x) r7(x)) and L(x,4) (fw) = Einn (x:),, .7 (x:)). respectively. Recall the definition of rademacher complexity w.r.t. S and
Rr, (E o H) E....e
Σ (h (x,) (xx))
sup NEH i=1
where El,...,, are independent rademacher random variables with (E. = 1) PPe -1) 1/2.
Then derive the following necessary lemma 45
46
Lemma 9. Let be a family of functions. For a loss function & bounded by μ. then any 0, with probability at least 1 5. for all h e H. such that
log 2/0
(h( s (s(h) + 2R. o H S) 3p
(30)
2n
and the theorem of Rademacher complexity of the method for loss function KL-divergence 8
Theorem 10. Let be a family of functions for multi-output linear regression, and H. be a family of functions for the j-th output. Rademacher complexity of ME(Maximumn-Enrre Entroppy with KL loss satisfies
RKL SF H. Ho < (g + +2 22) Σ Rn (H2 S).
(31)
Note that KL(u,) :) is not p-Lipschitz over Th for any p E R and u E R" Define function o(..)) as (T((:.F())) Next show that o(u,)) satisfy Lipschitzness. For p. R.
(é( p) - olu. q)l = IKL(u, (FL(),)F(p)) (L(u, SF(
28

---

which equals
Σ ui
exp (pi)
In
exp
(4)
E==1exp(p)
Σ j= exp (p,)
L=== exp (g:)
1 1+
c pi
1+
Σ
e
Ui.
j71
jft
Observing that In (1 + E, exp vs)
is 1-Lipschitz for E Imm thus right-hand side of preceding
equation bounded by
p 1. pi 9 8:112
1=1
|p 4/2 + Vc
lps 9.l.
< <vp + 1)I 9/12-
namely, i (vp 1)-Liischitz. According to 47 we have:
RKL SF Ho 5) V2( 1)
Σ
Ru (Hj o S),
j=1
9 Σ
= =VV2g v2)
Rn (H₂ S).
Although only implementing KL-divergence does not satisfy Lipschitzness, the combination of KL and Softmax(SF) is (yp + Lipsshhchi Define class of functions of j-th output with weight [88, rademacher complexity of H.
constraints Hj f w ||w 2l2 S $1}. According
satisfies
max4e(n) ||11|2
Ru (H o S)
Vn
Then right-hand handt-handaa side of Eq.
31 is bounded as
(v2g + v2)g
Rr (KL o SF H HoS) 5
max
|0:1|
(32)
Vn
xE(n]
According to Lemma 9 while we have the rademacher complexity of LDL with KL-diiergence loss defined in Eq. (32p. the bound can be defined as:
Lg (fw) - L(x,A) (fw) S (v2g
22)g
max |0lll (E(m
Vn
(33)
log 2/5
+ 3u
2n
Considering the node representations vary in different layers, Without loss of generality, we replace max;e(r] ||æall2 with max]e()).ve(]] x find the maximum node representation, instead of node feature. Then recall the PAC risk bound defined in Eq. 29
Lg (fw) - L(x,a) (fw)
122 log (og))))((W,)) + log
0o
22n
We use the sum both above bounds as the final risk bound for GCN with KL--divergence loss for the label distribution learning problem.
29

---

2(v2q V2)g
Lg (fm) ((x.A) (fr) S
max
x
vn
E !1]
1og2/6
+ 0
B2d 112 log -lo m1
3by
(34)
2n
7
As 49 suggests that 0 is replaced by a very small value, say 7 > O, for division by 0 when
implementing KL divergence, then for probability distribution p,q IR" with p; 7.9 2 Y
KL q) = Pr In
Σ 1=1
Pt
Pi In S -ln
1=1
thus there exists constant 2 - -n such that KL(,,)) s (e.g. b- 35 for y = 1 x 10-16).
Hence, all of the above concludes the proof.
H Supplement F: Additional Theoretical Analysis
H.1 Uniform Distribution Phenomenon
observed in the experiment that given fixed large hidden dimension f. normalized adjacency matrices are more easily to initialize approximate uniform attention than unnormalized adjacency matrices. the following, we provide theoretical analysis of the reason behind this phenomenon.
H.1.1 Normalized adjacency matrix are more likely to initialize uniform attention
For simplicity of analysis, we use one node's attentions from a single meta path graph in the study. Same analysis can be applied to all nodes and all meta-path graphs, without loss of generalizability.
Denote R"x" the adjacency matrix of the meta-path graph from meta-path P. and W e = the learnable weight of the meta-path graph (W is indeed Wi in Eq. (1) shown in the main manuscript. We drop Wh's subscript 0 and superscript i here for simplicity, as our analysis is based on a single meta-path graph). W!:. i] IRax denotes the column vector of W, and : I is row vector recording node i's connections (n the number of nodes in the meta-path graph).
for all the learnable matrices. d denotes A uniform weight initialization is with bounds [- - +
the average degree of the graph adjacency matrix.
Denote A and Á the adjacency matrix and normalized adjacency matrix of a graph, Theorem 11.
respectively. For node in the graph, its attention is proportional Ali, W[; O ( for the adjacency matrix, but proportional A[i,:] [[; O (77 for normalized adjacency matrix.
Proof. Normalization of A (ignoring the self-loop for simplicity) gives:
À + AD
(35)
1
Áli.i) =
Vd Ali
(36)
Denote Ej E Rinn a one-hot encoded vector with only jth position has value 1 (e.g.
(0, 1, 0, O, 0). Because ||ea|] 1, it's easy to know that e,, Vj can be used to construct an orthonormal eigenbasis. Denote E, as the index set of edges for node i. For example, if A[i,:] (0, 1, 1, 0, 0), then F. = (2, 3). because index position and has (i.e. edge connections) to node i. Then we can decompose Ali, :] and W[: as follows.
30

---

A[i.] Σ
Cj
(37)
JERs
A[i,:] Σ
ad
1
ej
(38)
JEE,
W i]
Σ
CLER
(39)
where is scalar for basis eg and is bounded by I- 1 v7 w/ Assume that eı & 0 for k E E; (meaning that the scalar is nonzero when nodes i and are connected), then have
d: Σ
: [[: = Σ
ej *
Σ
CKEK =
q ~
o(ddv/
(40)
JEE,
k
1=1
A[r. W[] = Σ
Ej Σ
CLCK
(41)
Vdi
JEE,
4
C1
O(L/VF)
(42)
d
where d; denotes the degree of node i (i.e. d, E A(i, :)) Compared to Eq. 42
vs. Eq. 40), it
easy conclude that plays more significant role in Eq. (42) Meaning that an unnormalized adjacency matrix affects the final attention with the average degree of the matrix while normalized adjacency matrix is dominated by the square root of hidden dimension f. With a relatively large hidden dimension, it is easy to see that normalized adjacency matrices are more likely to be dominated by the hidden dimension size f and produce uniform distribution of learned attention in the beginning stage.
H.2 Benefits of proposed attention structure in terms of label distribution learning
Similar structure of our proposed attention structure is applied when combining GNN and transformer structure, we provide an analysis on why such proposed attention structure is beneficial to graph label distribution learning. To show this, we first leverage the conclusion from Eq. (34) to obtain bound of graph label distribution learning on normal GCN layer.
B2d 1Pk log D W: +log af
There are three parts for the bounds, the part O
is our focus
7
as it is directly related to maximum degree d of graph adjacency matrix and is related to change in the GNN structure. It is easy to observe that our proposed attention structure follows the same GCN update except the adjacency matrix is now learnable from features and combined with topology space. So we assume that our model is a dynamic graph version and we focus on how the changes in the learnable graph could affect the bound. We show in the following that method reduces the maximum degree of graph adjacency matrix and therefore tightens the bound for graph label distribution learning.
Our attention layer learns a feature space topology and intersects its feature neighbor with its topology neighbor to get consistent neighbor set agreed by both feature and topology.
Theorem 12 (Equal priority of feature and topology space). Denote original graph structure as A, propagation graph for attention structure as A, denote 8( ((A(i,)) as the 7th node's degree. 2(A) as maximum degree of adjacency matrix A and arg max, 5 A [i, :] as the node index that has the maximum degree:
A(A) s ((A)
(43)
Proof.
Ay = ZQ(ZK)T
(44)
Ā = Softmar(A, A)
(45)
31

---

With @ operator, for any node i, it is easy to observe that
8(A(t,:)) ((A(t:))
(46)
Assume without loss of generality that A has (AA) at node j, i.e, arg max, ((A[i,]] j. and À has (AA) at node k,i.e, arg max, {(A : :]) = k. For arbitrary node in A,
( =I) - ( =)) - ((A
(47)
( = (() :)) < [k, :)) S -
(48)
We note that the equality sign only appears iff & j and ((A[k, :I) 8(A[k, :I) which relies on A at row & to be exactly the same as A row The probability is related to the consistency feature similarity and topology structure.
For graph with relatively low homophily score p; for a specific node with pj defined as
P4 NieNeighbor (Label
Label(t))
(49)
Neighhor
where Li)el(i) for a node i is the dominate class of its distribution and Neighbor(i) (i) is defined as topology neighbor set of node i. [Neiigborri) indicates the size of the set.
Assume that feature similarity strongly relates to label connection among nodes,i.e, features are similar between samples if their label distribution are close. The learnt As at row k satisfies the equality condition iff pą: 1. For practical graphs in general, such homophily score rarely exist and therefore our methods should reduce the maximum degree of the propagation graph in most cases.
Remark 1. With Theorem 6, we have shown that our proposed design (i.e using @ between learnt feature topology and graph topology) reduce maximum degree of the propagation graph with high probability. Leveraging derived bound term from Eq. (29) related to maximum degree, we can observe that lower maximum degree allows the model have tighter bound for HGDL problem in general. Therefore, our proposed design is naturally beneficial to the heterogeneous graph label distribution learning.
100 epochs (s)
DRUG (894)
URBAN (604)
ACM (5810)
YELP (3001)
DBLP (4057)
GCNKL
2.51
2.77
10.97
1.8569
23
HANKL
1.73
0.979
75
12.53
23.63
SeHGNNKL
2.0592
1.16
75.25
12.45
23.45
HGDLKL
1.19
1.05
8.81
8.75
26.57
Table 14: Runtime results for 100 training epochs. The numbers of target nodes are labeled beside their corresponding dataset names.
H.3 Runtime Discussion
To verify the efficiency of our method, Table 14 shows the runtime for 100 training epochs for our baseline methods, Each method has been trained with 100 epochs. We can observe that our proposed HGDL algorithm enjoys good scalability by attaining runtime results that comparable to the baseline GCNs. Compared to other baseline such as SeHGNN, HGDL is in general much faster. This indicates that method is more efficient for learning heterogeneous networks. This can be mainly attributed to the automated learning of weights to control individual meta-paths and then integrates them to yield optimal graph topology homogenization for label distribution learning.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our study aims generalize Label Distribution Learning from IID data the non-lID, networked data, and we have made this claim clear in the Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed the limited performance gain on CAD and CLD metrics in Section 6.3 and provided Runtime analysis in Supplement Section H.3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
33

---

Answer: [Yes]
Justification: We have analyzed the generalization error bound the proposed algorihtm in PAC-Bayes regime and provided the complete proof in Supplement D and E.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
[Yes]
Justification: We have provided tuned parameter settings in provided Github link and reported their impact empirical results in Supplement C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
34

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have uploaded the code and benchmark datasets as the supplementary submission provided Github link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public/guides httpss/eooeeeueoaaiooooooooeoeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results, See the NeurIPS code and data submission guidelines (https //nip..c. wttp:ccccppcccccceeceec gww /eodedeee/ooeebbiieeeeee wwttimiiiioonnniiieeooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have presented the creation and preprocessing steps of the benchmark datasets and the implementation details of our algorithms in Supplementary and C, respectively.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
We generated experimental results from repeated experiments (including different train/test split) and leveraged t-test validate the statistical significance in settings where our algorithm outperform its competitors. We reported its win/tie/loss in Section 6.2 where "win" indicates that our algorithm outperforms the compared models with confidence level at level of 90%.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
35

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We disclosed our computing node setup. Note, our algorithm is not computa- tional demanding and could be generalized onto various single-GPU machines.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We revised the COE and all agreed on it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
36

---

Justification: We demonstrated multiple potential application of our algorithm with the benchmark dataset, including the use case of Urban Functionality Delineation, which has been leveraged as the motivating example of our proposal in Introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
37

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: The artifacts generated from this paper will be published under CC-BY 4.0 license for public access and non-commercial use.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
38

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "LNlcXoIj3M", "reviewer": "Reviewer_FJkr", "summary": "This paper advances Label Distribution Learning (LDL) into the realm of graph domains, specifically addressing the heterogeneous graph label distribution learning (HGDL) problem. The authors highlight that graph heterogeneity, reflected in node types, node attributes, and neighborhood structures, poses significant challenges for generalizing LDL to graphs. To tackle these challenges, the authors propose a new learning framework with two key components: \n1.Proactive Graph Topology Homogenization: This component focuses on learning optimal information aggregation between meta-paths to address node heterogeneity before the embedding learning phase.\n2. Topology and Content Consistency-aware Graph Transformer: This component uses an attention mechanism to learn the consistency between meta-paths and node attributes, ensuring that network topology and nodal attributes are equally emphasized during label distribution learning.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The introduction of a framework that proactively addresses graph heterogeneity and incorporates consistency-aware mechanisms is a novel contribution to the field.\n\n2.The use of KL-divergence and additional constraints in an end-to-end learning process provides a robust solution for label distribution learning on graphs.\n\n3.The theoretical and experimental validations are thorough, and the availability of code and datasets promotes transparency and reproducibility.", "weaknesses": "1. The paper lacks an analysis of the algorithm's complexity, and the framework diagram in Figure 2 is somewhat cluttered.\n\n2. The authors' motivation is to address the challenges posed by graph heterogeneity, but these challenges are not clearly described in the introduction.\n\n3. For optimization formula (3), the authors need to provide a detailed explanation of the advantages of this design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We address the reviewer's concerns below one by one in a Q&A fashion.\n  \n  ---\n\n\nQ1: The paper lacks an analysis of the algorithm's complexity.\n\nA1: We have conducted the complexity analysis, which has been deferred to the Appendix H.3 due to page limits. We briefly summarize the results of our analysis for the reviewer's convenience. Specifically, our method involves learning a new adjacency matrix from all meta-paths. Without considering any decomposition methods, the number of learnable parameters has to be at least $\\mathcal{O}(n)$ for the graph topology homogenization stage, where $n$ denotes number of nodes. So by the attention mechanism, our proposed architecture entails a number of $knf+k^{2}f$ parameters, where $k$ is the number of meta-paths used and $f$ is the dimension of latent space. Omitting constants this amounts to an $\\mathcal{O}(n)$ number of parameters to be learned. To prepare the feature adjacency matrix that mitigates feature-topology inconsistency, it requires a number of learnable parameters $h^2 + mh$, where $h$ and $m$ represent the dimensions of node embedding and input nodal content, respectively. This reduces to a constant $\\mathcal{O}(1)$ by screening out factors $h$ and $m$, which are relatively neglectable compared to $n$.\n\nOverall, the number of learnable parameters of our proposed model is $\\mathcal{O}(n)$, which seemingly scales up with the total number of nodes in a linear way. However, we note that such linearity is not necessary if the message-passing at each iteration is conducted among a subset of node, instead of all nodes, such as through sampling as done by GraphSage. As such, we can further reduce the computation complexity without being bottlenecked by sizeable graphs with large number of nodes. \n\n  ---\n\nQ2: Figure 2 somewhat cluttered.\n\nA2: We have improved the visual clarity of Figure 2 and put it into the pdf. file as \"Figure 3\". \n\n  ---\n\nQ3: The challenges posed by graph heterogeneity have not been clearly described in the introduction.\n\nA3: We will enrich the two challenges using the motivating example shown in Figure 1 in the manuscript. Specifically, we will correlate the challenges to the URBAN dataset as follows. Challenge (1): Graph heterogeneity that complicates the message-passing between nodes of a specific type, as the label distributions of those nodes are influenced by their neighboring nodes that may vary in terms of types, content, and topological features. For example, R3 and R4 are two immediately neighboring residence regions but are connected to different functional areas, i.e., R3 connects to Leisure and Service areas but R4 connects to Service region only. Thus, the urban functionality (i.e., label distribution) of R3 and R4 eventually differs. \n\nChallenge (2): Graph topology and nodal features that may suggest inconsistent label distributions, where nodes sharing similar contents are positioned far apart and separated by various nodes of different types on the graph. Unlike traditional LDL that focuses on instance vectors only, an effective LDL model on graphs require harmonizing nodal contents with topological structures for a unified representation.  For example, R2 to topologically more faraway from R3 compared to R4; however, comparing their nodal contents, local neighborhood structures, and functionality distributions, R2 is more similar to R3 over R4.  \n\nWe will add this motivating example in our camera ready.\n\n  ---\n\nQ4: For Eq. (3), please explain the advantages of this objective design.\n\nA4: We rationale our design as follows. The optimization objective in Eq. (3) has two KL divergence terms. (1) The first term minimizes $D_{\\text{KL}}(Y \\| \\hat{Y})$, enforcing the predicted label distributions to be similar to the ground-true distributions (if exist). (2) The second maximizes the distance between the weight distribution for all meta-paths $\\Theta[i, :]$ and a uniform distribution $U[1, k]$. \nThe intuition is to encourage the diversity of the learned weights associating with different meta-paths. The reason is that KL divergence naturally favors uniform distribution on small values, encouraging all weights to be learned similar. The second term is to counter against this: the noisy meta-path that has been ill defined will diminish its negative impact on the model performance, whereas other meta-paths resulting in better performance will be promoted. This ensures that the important meta-path information will dominate the resultant adjacency matrix, leading to optimal meta-path topology harmonization. \n\nIn addition to the intuitions, we present the sensitivity analysis on the hyperparameter $\\gamma$ as empirical evidence.  $\\gamma$ controls the impact of the second term of Eq. (3). We illustrate the trends of KL divergence between the predicted and true label distributions w.r.t. the value of $\\gamma$ on all five datasets in Figure~1. Specifically, different values of $\\gamma$ have been applied, including 0, 1e-5, 1e-4, 1e-3, 1e-2, and 0.1. We observe that the lowest KL divergence value occurs at various $\\gamma$ choices, e.g., the optimal KL divergence occurs on the Drug and URBAN datasets at $\\gamma = 0.1$, YELP and DBLP dataset at $\\gamma = 1e-3, 1e-5$, respectively. The non-zero values of $\\gamma$ necessitates the design of the second term in Eq. (3). \n\nWe will supplement the intuition of the design of Eq. (3) and include the sensitive analysis on $\\gamma$ in our camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bD4tuGUZNN", "reviewer": "Reviewer_EYc1", "summary": "This paper introduces a novel approach to Label Distribution Learning (LDL) specifically tailored for heterogeneous graphs, addressing the inherent complexities and challenges associated with this domain. By highlighting the necessity of LDL in heterogeneous settings and outlining the unique challenges involved, the paper lays a foundation for advancing research in this emerging field.\nThe proposed method is underpinned by a robust theoretical framework, providing a coherent rationale for its design and implementation. \nExperimental validation across five distinct datasets and evaluation using six metrics demonstrate the method's effectiveness over established baselines.", "questions": "I don't have any questions.", "limitations": "While the authors provide a complexity analysis and mention that scalability is not the main concern of this paper, they acknowledge that the current model may not scale well with larger datasets. Although a modification to improve scalability is suggested, it is left for future work, indicating that the current version might struggle with large-scale data.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper is the first to identify the necessity of Label Distribution Learning (LDL) on heterogeneous graphs, addressing the unique challenges and complexities associated with this task. \n2.The proposed method is well-grounded in theory, providing a solid foundation for its design and implementation. The authors present a clear and thorough theoretical framework that supports the efficacy and rationale behind their approach, enhancing the credibility and robustness of the method.\n3.The effectiveness of the proposed method is demonstrated through extensive experiments on five diverse datasets and six different metrics. The results consistently show that the method outperforms the baselines, indicating its superiority and practical applicability across various scenarios and evaluation criteria.", "weaknesses": "1.The paper categorizes current LDL methods into three distinct types in the related work section. However, the experimental evaluation only includes three models (GCN, HAN, and SeHGNN) with the KL-divergence loss function as baselines. This limited selection raises questions about whether the experiments sufficiently demonstrate the proposed method's superiority over the broader range of existing LDL methods.\n2.The study employs γ as a hyperparameter in the proposed model but does not provide any analysis of how variations in this parameter affect the model's performance. A thorough analysis of the hyperparameter's impact would offer valuable insights into the model's sensitivity and robustness.\n3.The paper does not present a detailed comparison of the computational time required for the proposed method versus the baselines. Without this information, it is unclear whether the proposed method is more efficient in terms of runtime.\n4. The motivation of this paper is not very convincing. It seems that this work is done just because LDL has not been applied in heterogeneous graph. It is not clear why LDL can solve some key problems in heterogenous graph.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We address the reviewer's concerns below one by one in a Q&A fashion.\n\n--- \n\nQ1: Limited comparison to the existing LDL methods.\n\nA1: We would like to kindly point out that the existing LDL methods were not tailored for handling graphs (networked data). In fact, GLDL is the only method specifically designed for label distribution learning on graph data, yet has been restricted to homogeneous network. \n\nWe have supplemented a comparative study with GLDL on the DBLP and DRUG datasets, with results presented in the Table~1 of the uploaded pdf file. The results indicate that GLDL outperforms other baselines but is still inferior to our proposed HGDL method. This is mainly because that, although  GLDL and HGDL both considered label distribution in their learning processes, HGDL individualizes the learning of weights associated with different meta-paths and integrates them to construct optimal graph topology homogenization. In contrast, GLDL only generates one single meta-path for LDL by heuristics, which may end up with suboptimal solutions. \n\nWe will include these comparative results with GLDL and supplement a discussion in our camera-ready.\n\n--- \n\nQ2: Sensitivity analysis on the hyperparameter $\\gamma$.\n\nA2: We present the sensitivity analysis on the hyperparameter $\\gamma$, which controls the impact of the second term of Eq. (3). We illustrate trends of KL divergence between predicted and true label distributions w.r.t. values of $\\gamma$ on all five datasets in Figure~1. Specifically, different values of $\\gamma$ have been applied, including 0, 1e-5, 1e-4, 1e-3, 1e-2, and 0.1. We observe that the lowest KL divergence value occurs at various $\\gamma$ choices, e.g., the optimal KL divergence occurs on the DRUG and URBAN datasets at $\\gamma = 0.1$, and YELP and DBLP dataset at 1e-3 and 1e-5, respectively. The non-zero values of $\\gamma$ necessitates design of the second term in Eq. (3).  We will include the sensitive analysis on $\\gamma$ in our camera-ready.\n\n--- \n\nQ3: A comparison of the computational time to show the efficiency of the proposed approach against baselines. \n\nA3: We have completed a runtime comparison as presented in Table 2 in the uploaded PDF file. Each method has been trained with 100 epochs. Note, due to various neural architecture designs, different models may take varied number of training epochs to converge. Thus it has become a norm to compare their runtime performance on a fixed number of epochs (or per epoch training time), which has also been done in [1]. We can observe that our proposed HGDL algorithm enjoys good scalability by attaining runtime results that are comparable to the baseline GCNs. Compared to other baseline such as SeHGNN$_{KL}$,  HGDL is in general much faster. This indicates that our method is more efficient for learning heterogeneous networks. This can be mainly attributed to the automated learning of weights to control individual meta-paths and then integrates them to yield optimal graph topology homogenization for label distribution learning.  We will add the runtime comparison in our camera-ready.\n\n[1] Hao Yuan, Yajiong Liu, Yanfeng Zhang, Xin Ai, Qiange Wang, Chaoyi Chen, Yu Gu, and Ge Yu. \"Comprehensive Evaluation of GNN Training Systems: A Data Management Perspective,\" VLDB 2024.\n\n--- \n\nQ4: The motivation of the research should be further clarified. Why LDL can solve the key problems imposed by heterogenous graph?\n\nA4: Indeed our research is mainly motivated by the fact that no LDL model has been tailored for heterogeneous graph.\nWe note two prominent challenges imposed by the network heterogeneity that motivate our research, as follows. First, in heterogeneous graph, the label distribution of each node is influenced by their neighboring nodes that can vary through multiple factors, including their different types of nodes and edges, nodal contents, and topological features. This complicates the message-passing mechanism, \nbecause aligning those varying factors would result in a combinatorial issue. Second, nodes sharing similar contents may be frequently positioned far apart in heterogeneous graphs, separated by nodes of other types, resulting in substantial topological distances between them. During message-passing, the impact of distantly positioned nodes within a graph is substantially diminished, consequently steering the LDL model to prioritize individual nodal vectors, overlooking graph topology. \n\nOur LDL research targets to resolve the heterogeneous graph challenges with two building blocks. First, we propose to use multiple weighted meta-paths in addition to regular KL divergence loss for class distribution learning. The weights are learned, allowing each meta-path to individualize its message-passing paths without being negatively impacted by the different node and edge types. Second, HGDL uses a consistency-aware graph transformer architecture to harmonize local topology and global feature information. The graph transformer aligns nodal features with the learned optimal topology through weighted meta-paths, capturing both local neighborhood information and global feature similarities. This harmonization is crucial for ensuring that nodes with similar features, even if topologically distant, are represented in a way that reflects their content-based similarities.\n\nIn addition, from a data perspective, the label distribution is often more informative than a single class scalar. Consider our motivating example in the manuscript, delineating the local urban region with a distribution over multiple city functional classes will lead to richer understanding of the composition of the particular region, such as the number of buildings for various civil purposes, including housing, healthcare, education, etc. Examples of such application abound, including business network (YELP), citation network (ACM), protein-disease network (DRUG), with their datasets all studied in our experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CbSPgawUoz", "reviewer": "Reviewer_pXs2", "summary": "This paper studies the problem of heterogeneous graph label distribution learning. To deal with it, this paper proposes an HGDL method that optimizes meta-path graph topology and aligns it with nodal features for consistent message-passing, backed by theoretical support. Experimental results on five datasets demonstrate the effectiveness of the proposed model.", "questions": "Q1. Why only search for meta-paths connecting nodes of the target type rather than aggregating information from neighbors of different types?\n\nQ2. Why can the design in Section 4.2 effectively capture global feature consistencies? Could you provide more detailed explanations?\n\nQ3. Could you provide an explanation for why HGDL performs worse on the CLD metric?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1. This paper is the first to investigate the LDL problem in heterogeneous graphs, which seems interesting.\n\nS2. HGDL effectively combines meta-path aggregation and transformer-based methods to ensure consistent node label distribution learning, validated by both theoretical analysis and empirical studies.\n\nS3. The contributions are clearly written, with codes and datasets provided for reproducibility and practical utility.", "weaknesses": "W1. The intuitions behind the techniques lack explanation. More details can be found in Q1 and Q2.\n\nW2. The approach section is somewhat difficult to follow and could benefit from improved presentation.\n\nW3. It would be beneficial to include baselines for the LDL problem, such as GLDL [1]. While these methods are not specifically designed for heterogeneous graphs, it is still important to treat heterogeneous graphs as homogeneous ones and apply these methods for comparison. This will provide a more comprehensive evaluation of the proposed approach.\n\nW4. This paper compares only three baselines, which are relatively dated for heterogeneous graph learning. For example, GCN was published in 2017, HAN in 2019, and SeHGNN in 2023. This makes it challenging to convincingly demonstrate the advantages of the proposed HGDL. Thus, it is essential to include more recent baselines, such as HINormer [2].\n\nReferences:\n\n[1] Y. Jin, R. Gao, Y. He, and X. Zhu, “Gldl: Graph label distribution learning,” in Proceedings of the 38th Annual AAAI Conference on Artificial Intelligence, 2024.\n\n[2] Qiheng Mao, Zemin Liu, Chenghao Liu, and Jianling Sun. \"Hinormer: Representation learning on heterogeneous information networks with graph transformer\", in Proceedings of the ACM Web Conference, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We address the reviewer's concerns below one by one in a Q&A fashion.\n\n---\n\nQ1: Why only search for meta-paths connecting nodes of the target type rather than aggregating information from neighbors of different types?\n\nA1: We justify our meta-path aggregation design as follows. Assume we have types T, A, and B, with T being the target node type. We assume that the reviewer was asking: why we only consider meta-path type like T-T, T-A-T, T-A-B-A-T, etc., but not consider information aggregation via other paths such as A...A, B...B, T..A..B, etc. We answer this question from two perspective. \n\nFirst, for heterogeneous network with different types of nodes, each node type may have its own nodal feature space. Aggregation information from different node types is a major challenges in this setting. One may think of treating all node types equally by mapping those different feature spaces onto one shared latent subspace, on which the aggregation can be carried out. However, this solution would require a very high-dimensional adjacency matrix that encodes the graph topology consisting of all types of nodes, with dense node connectivity, making it both computational and memory demanding. In contrast, our proposed solution leverages meta-path to yield homogeneous graph of the target type only. This results in sparse meta-path graphs, alleviating computational overhead and preventing oversmoothed node embeddings during the message-passing [1]. \n\nSecond, in our LDL modeling, the main objective is to predict label distributions for target nodes. The meta-paths starting from and ending with the target nodes provide a clear model of  how the information within the neighborhood substructures of those target nodes can be aggregated. Other paths that do not entail the target node such as ABA, BAB, etc., do not serve for this goal directly, and there could be many such meta-paths, resulting in combinatorics numbers that makes our model overly complicated. Despite that exhausting all such meta-path combinations is in general possible, the homogeneous graphs obtained from them will again incur inconsistent feature spaces, e.g., consider two graphs resulted from T-A-T and A-A, of which the feature dimensions equate to the dimensions of T and A node types, respectively. Merging these graphs is still challenging. \n\n\n[1] Kenta Oono, & Taĳi Suzuki. Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. ICLR 2020.\n\n---\n\nQ2: Why can the design in Section 4.2 effectively capture global feature consistencies?\n\nA2: In our paper, the global feature consistency refers to a regularization that ensures the nodes with similar original feature inputs, albeit being topologically faraway, should have similar embedding vectors. To have an explicit model on this, we draw insights from the attention mechanism in graph transformers to construct a feature adjacency matrix $(ZQ) (ZK)^{\\top} \\in \\mathbb{R}^{n \\times n}$, as stated in Eq. (2). Intuitively, the attention scores over nodes of similar content are larger, compared to those nodes with unrelated contents. The Hadamard product between this feature adjacency matrix and the weighted meta-path graph $\\tilde{A}$ will balance the (global) feature consistency and (local) topology consistency. We shall add more justification regarding our design in Section 4.2 in our camera-ready version.\n\n\n---\n\nQ3: Could you provide an explanation for why HGDL performs worse on the CLD metric?\n\nA3: This is mainly because of the inconsistency between the KL Divergence and Clark distance (CLD) measurements, with the same phenomenon observed and reported in Table 2 of [2]. Specifically, we can make a concrete example that demonstrates this inconsistency. For instance, let y = [0.01,0.01,0.98], $\\hat{y_{1}}$ = [0.05,0.05,0.9], and $\\hat{y_{2}}$=[0.03,0.07,0.9], where $\\hat{y}_1$, $\\hat{y}_2$ indicate the predicted results from model1, model2, respectively. (KL divergence, CLD) of Model1 and Model2 are (0.4467,0.0222) and (0.2528,0.023), respectively, implying a tradeoff region. Generally, there exists a tradeoff region in the small probability values between Clark distance and KL divergence. We visualize this tradeoff in Figure~2 in the uploaded PDF file. In the figure, the black and blue curves delineate the loss trends of KL divergence and CLD, respectively. We can observe that tradeoff region of the two loss function falls in range [0.019, 0.1] along x-axis. This tradeoff explains the cases where HGDL does not attain superior CLD performance but still excels in terms of KL divergence. In general our HGDL algorithm outperforms the competitors in both KL and CLD in most cases. \n\n[2] X. Geng, \"Label Distribution Learning,\" in IEEE Transactions on Knowledge and Data Engineering, vol. 28, no. 7, pp. 1734-1748, 2016.\n\n---\n\nQ4: Need a comparison with GLDL.\n\nA4: We have completed the comparative study and presented the results in the Table~1 of the uploaded pdf file. Please refer to A1 to the Reviewer EYc1 for more details.\n\n---\n\nQ5: Include more recent baselines such as HINormer.\n\nA5: We have carried out the experiments on DRUG and DBLP dataset using HINormer as an additional competitor. Results are presented in Table~1 in the uploaded PDF file. We only use target node features to level the comparison, which has been applied to all baselines. Due to time limitation,  we searched the hyperparameter of hidden dimension in [32,64,128,256], $\\beta$ in [0.5,1], temperature in [0.5,1], and the number of GNN layers in [2,3]. We can observe that HINormer does not outperform other baselines. One reason might be the inconsistency of node feature aggregation and the relation encoding scheme. The node vectors are set to zero other than the target nodes, according to original code provided by the HINormer authors. Simply replacing BCE with KL-divergence does not make HINormer well for LDL. We will include the results in camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VRqlumywcY", "reviewer": "Reviewer_xSZH", "summary": "This paper studies heterogeneous graph label distribution learning with the aim of predicting label distributions of unlabeled nodes in a heterogeneous graph. This paper elaborates the challenges for generalizing LDL into networked data, and proposes an LDL algorithm HGDL to overcome the challenges. Besides, this paper derives the PAC-Bayes error bound for HGDL and conducts experiments to show the superiority of the proposal.", "questions": "1.\tWhy can't we just replace the output layer of existing heterogeneous node classification model with MSE (Mean Squared Error) or KL loss to learn the label distribution? What challenges arise when replacing categorical labels with label distributions in HGDL? And which of these problems does this paper address? What is the contribution of this paper to the field of label distribution learning?\n2.\tWhy use PAC-Bayes theory to analyze the error bound instead of PAC theory?", "limitations": "I don't believe that the paper has a potential negative social impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper studies a new problem, i.e., label distribution learning in heterogeneous graphs. Besides, this paper proposes an end-to-end HGDL learning approach to jointly learn the optimal meta-path graph topology and node representation. The effectiveness of the proposed method is studied theoretically and empirically.", "weaknesses": "The contribution of this paper is unclear. This paper attempts to combine heterogeneous graph learning with label distribution learning. However, the paper addresses challenges about the topology, heterogeneity, and inconsistency in terms of instances, and pays little attention to the challenges of learning label distributions of instances, i.e., the proposed challenges do not arise from label distributions. Therefore, the contributions for LDL are not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We address the reviewer's concerns below one by one in a Q&A fashion.\n\n--- \n\nQ1: Why can't we just replace the output layer of existing heterogeneous node classification model with MSE (Mean Squared Error) or KL loss to learn the label distribution? \n\nA1:  Simply replacing the output layer with MSE or KL loss to learn the label distribution, like the reviewer has suggested, will not work well due to two prominent challenges imposed by the network heterogeneity. First, in heterogeneous graph, the label distribution of each node is influenced by their neighboring nodes that can vary through multiple factors, including their different types of nodes and edges, nodal contents, and topological features. This complicates the message-passing mechanism, because aligning those varying factors would result in a combinatorial issue. Second, nodes sharing similar contents may be frequently positioned far apart in heterogeneous graphs, separated by nodes of other types, resulting in substantial topological distances between them. During message-passing, the impact of distantly positioned nodes within a graph is substantially diminished, consequently steering the LDL model to prioritize individual nodal vectors, overlooking graph topology.\n\nOur LDL research targets to resolve the heterogeneous graph challenges with two building blocks. First, we propose to use multiple weighted meta-paths in addition to regular KL divergence loss for class distribution learning. The weights are learned, allowing each meta-path to individualize its message-passing paths without being negatively impacted by the different node and edge types. Second, HGDL uses a consistency-aware graph transformer architecture to harmonize local topology and global feature information. The graph transformer aligns nodal features with the learned optimal topology through weighted meta-paths, capturing both local neighborhood information and global feature similarities. This harmonization is crucial for ensuring that nodes with similar features, even if topologically distant, are represented in a way that reflects their content-based similarities.\n\n--- \n\nQ2: What is the contribution of this paper to the LDL research?\nA2: From the label distribution learning perspective, our research conveys four key contributions as follows. \n\n(1) Our research is the first to generalize the LDL to heterogeneous networks. The practical implications such as for urban functionality prediction or drug function prediction (presented in Sec 6.1) and the theoretical analysis, to our knowledge, has not yet been explored by any contemporary research.\n\n(2) Our research provides a simple yet effective way of modeling message-passing in heterogeneous network for label distribution learning. It also offer transparent interpretation to explain which meta-path play a more important role for the final outcome.\n\n(3) Our theoretical study not only offers assurance of the proposed model performance, but also paves the way to enrich theoretical understanding of label distribution learning for networked data.\n\n(4) We have created new datasets to validate the algorithm performance. Both our data and algorithms are published to stimulate future growth of research in label distribution for networks, which has been thus far under-explored. \n\n--- \n\nQ3: Why use PAC-Bayes theory to analyze the error bound instead of PAC theory? \n\nA3: We analyzed the algorithm performance in a PAC-Bayes regime instead of PAC for three main reasons. (1) Traditional PAC learning theory focuses on a single, deterministic model/hypothesis, which often leads to worst-case analysis. In contrast, PAC-Bayes extends it by integrating a Bayesian perspective, offering probabilistic bounds over a distribution of hypotheses. PAC-Bayes gauges how well a learner can generalize from a prior (model-dependent) to a posterior (data-dependent), thereby providing insights into the learning process that are not captured by PAC theory alone. This often results in tighter and more informative generalization bounds, especially for parameter-rich models such as neural networks [1].\n\n(2) GNNs are inherently complex to model graph data with non-linear relationships. The high dimensionality and complexity of GNN models make PAC-Bayes particularly suitable for their analysis, as also indicated by [2]. In particular, PAC-Bayes lends insights into the interplay of model choice (prior) and data observation (posterior), allowing for the integration of specific model and data properties into the resultant bounds. Thanks to this, our analysis can establish a correlation between the generalization error and the maximum degree of the searched meta-path graph $\\tilde{A}$ in HGDL, as detailed in Theorem 2. Traditional PAC theory that mainly makes the underlying data IID assumption lacks the capacity to incorporate the properties of graph data into analysis.\n\n(3) PAC-Bayes leverages KL-divergence as an information-theoretic metric, which is well-suited for measuring distances in continuous spaces, such as those encountered in our LDL modeling. Our approach to quantifying discrepancies between label distributions relies on KL-divergence, making PAC-Bayes a natural fit for our analysis. This alignment simplifies our derivation and ensures that our theoretical insights aligns with the empirical results.\n\n\n[1] Neyshabur, Behnam, Srinadh Bhojanapalli, and Nathan Srebro. \"A pac-bayesian approach to spectrally-normalized margin bounds for neural networks.\" ICLR 2018.\n\n[2] Liao, Renjie, Raquel Urtasun, and Richard Zemel. \"A pac-bayesian approach to generalization bounds for graph neural networks.\" arXiv 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their positive and constructive comments. Here, we summarize the major concerns and our responses.

(1) Add GLDL and HINormer as new rival models (suggested by Reviewers EYc1 and pXs2). 

- We have added new comparative results with the two models, presented in Table 1 in the uploaded PDF file. From the results, we can observe that our HGDL model still outperforms GLDL and HINormer in our benchmark datasets. 

(2) Add sensitivity analysis on the hyperparameter $\gamma$ (suggested by Reviewers FJkr and EYc1).

- We have conducted this sensitivity analysis, with results illustrated in Figure~1 in the uploaded PDF file. We note that the optimal choice of $\gamma$ varies across different datasets, which substantiates the impact of the regularization term in the overall objective design. 

(3) Add runtime comparison to evaluate the algorithm efficiency (suggested by Reviewers FJkr and EYc1).

- We have compared the runtime performance of our algorithm and its compared methods, with results documented in Table~2 in the uploaded PDF file. The results show that the efficiency performance of our HGDL approach is on a par with the GCN and HAN methods and is faster than the graph transformer including SeHGNN models.

In below, we have provided rebuttal to each individual reviewer to address their concerns in a Q&A fashion. If there are any points that require further clarification or additional information to facilitate the decision-making process, please do not hesitate to let us know. We are more than willing to provide further explanations or details.

Meta-review: This paper advances label distribution learning (LDL) into the realm of graph domains, specifically addressing the heterogeneous graph label distribution learning (HGDL) problem. The authors propose an HGDL method that optimizes meta-path graph topology and aligns it with nodal features for consistent message-passing. The proposed method is underpinned by a robust theoretical framework, providing a coherent rationale for its design and implementation. Experimental validation across five distinct datasets and evaluation using six metrics demonstrate the method's effectiveness over established baselines. The novelty and contribution of this paper have been acknowledged by all the reviewers. Thus, I would recommend accepting this paper.
Decision: Accept (poster)