ATTRIBUTES RECONSTRUCTION IN HETEROGENEOUS NETWORKS VIA GRAPH AUGMENTATION
Authors: yixuan Liang
yuan wan
Venue: Submitted to ICLR 2023 (2023)

Abstract: Heterogeneous Graph Neural Networks(HGNNs), as an effective tool for mining heterogeneous graphs, have achieved remarkable performance on node classification tasks. Yet, HGNNs are limited in their mining power as they require all nodes to have complete and reliable attributes. It is usually unrealistic since the attributes of many nodes in reality are inevitably missing or defective. Existing methods usually take imputation schemes to complete missing attributes, in which topology information is ignored, leading to suboptimal performance. And some graph augmentation techniques have improved the quality of attributes, while few of them are designed for heterogeneous graphs. In this work, we study the data augmentation on heterogeneous graphs, tackling the missing and defective attributes simultaneously, and propose a novel generic architecture—Attributes Reconstruction in Heterogeneous networks via Graph Augmentation(ARHGA), including random sampling, attribute augmentation and consistency training. In graph augmentation, to ensure attributes plausible and accurate, the attention mechanism is adopted to reconstruct attributes under the guidance of the topological relationship between nodes. Our proposed architecture can be easily combined with any GNN-based heterogeneous model, and improves the performance. Extensive experiments on three benchmark datasets demonstrate the superior performance of ARHGA over strate-of-the-art baselines on semi-supervised node classification.

Under review as a conference paper at ICLR 2023
ATTRIBUTES RECONSTRUCTION IN HETEROGENEOUS NETWORKS VIA GRAPH AUGMENTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Heterogeneous Graph Neural Networks(HGNNs)) as an effective tool for mining heterogeneous graphs, have achieved remarkable performance on node classifica- tion tasks. Yet, HGNNs are limited in their mining power as they require all nodes to have complete and reliable attributes. It is usually unrealistic since the attributes of many nodes in reality are inevitably missing or defective. Existing methods usually take imputation schemes to complete missing attributes, in which topol- ogy information is ignored, leading to suboptimal performance. And some graph augmentation techniques have improved the quality of attributes, while few of them are designed for heterogeneous graphs. In this work, we study the data aug- mentation heterogeneous graphs, tackling the missing and defective attributes simultaneously, and propose a novel generic ciantecture-Attibbuuee Reconstruc- tion in Heterogeneous networks via Graph Augmentation(ARRHGA. random sampling, attribute augmentation and consistency training. In graph aug- mentation, to ensure attributes plausible and accurate, the attention mechanism is adopted to reconstruct attributes under the guidance of the topological relation- ship between nodes. Our proposed architecture can be casily combined with any GNN-based heterogeneous model, and improves the performance. Extensive ex- periments on three benchmark datasets demonstrate the superior performance of ARHGA over strate-of-the-art baselines on semi-supervised node classification.
including
INTRODUCTION
Heterogeneous information networrs((IIs))\angee et al. (2020); (Shi al. (2016): Shen al. (017)) ).
which contain multiple types of nodes and edges, have been widely used to model complex systems and solve practical problems. Recently, heterogeneous graph neural networks have emerged as
prevalent deep learning architectures to analyze HINs and shown superior performance in various graph analytical tasks, such as node classification Wang et al. (2019)); Yun et al. (2019)) and link prediction et al.] (2020); Zhang et al. (2019). Most HGNNs follow message message-passing scheme in which each node updates its embedding by aggregating information of its neighbors' attributes. Such message-passing scheme usually requires that all nodes have complete and reliable attributes, which is not always satisfied in practice due to resource limitation and personal privacy, resulting in missing and defective attributes.
In general, the attribute missing in heterogencous graphs means that attributes of partial nodes are entirely missing, compared that in homogeneous graphs, is more frequent and complex. Take DSLP & Han (2013) as an example, the network has four types of nodes(author, paper, term and venue) and three types of links. Only paper nodes have attributes which are extracted from the keywords in their titles, while other types of nodes have no attributes. It impairs the effectiveness of the corresponding graph mining model to certain extents. In another fold, the original attributes of nodes sometimes not ideal since heterogeneous graphs extracted from complex systems which inevitably are subject to various forms of contamination, such as mistakes and adversarial attacks, making error propagation and greatly affecting the process of message-passing This sug- gests the need for effective approaches able to complete missing attributes and calibrate defective attributes in heterogeneous graphs simultaneously.
To alleviate the effect incurred from missing attributes, the existing models usually adopt imputation strategy, such as neighbor's average or one-hot vector as done in MAGNN et al. (2020)) These

---

Under review as a conference paper at ICLR 2023
imputation methods nonoptimal because graph structure information is ignored and only rare useful information is provided, hampering subsequent amalysis. An alternative technique to tackle this issue is to consider graph topology information and inject it into the completion models. The work of Jin et al. (2021) and He et al. )002)) has shown a significant boost on node classification tasks. But both methods naturally assume that the original attributes to satisfy in real-world applications. In another concern of research, some graph augmentation tech- niques are adopted calibrate original attributes improve the quality and have shown a promising performance Xu et al. ) Zhu et al. (2021)) However, these methods are deficient for hetero- geneous graphs as they are not capable of encoding complex interaction. Further, existing methods either only complete missing attributes or only improve the quality of attributes, while it is worthy making efforts to solving both problems at the same time.
reliable, which is not easy
In this paper, we attempt to deal with the missing and defective attributes simultaneously in hetero- geneous graphs, and propose a novel framework for Attributes Reconstruction in Heterogeneous networks via Graph (ugmentation(ARRGA)) ARHGA repeatedly sample nodes to perform at- tribute augmentation to obtain multiple augmented attributes for each node, and then utilize con- sistency training(XI) Xie et al. (2020)) to make the outputs of different augmentations as similar as possible. Moreover, to ensure the augmented attributes more accurate, node topological embed- dings are learned through HIN-embedding methods metno
al. (2017); Fu et al. (2017); [Shang] et al. (2016); Waan et al. (2019b)) to capture graph structure information as guidance. In this way, ARHGA effectively enhances the performance of existing based heterogencous models in aid of the reconstructed attributes.
Contributions. In summary, the main contributions of this paper are as follows:
We propose generic architecture of graph augmentation on heterogeneous networks for attributes reconstruction, focusing both on the missing and defective attributes.
We design an effective attribute-wise augmentation strategy implemented by attention mechanism, which integrates topology information to increase the reliability of the recon- structed attributes.
Extensive experimental results on three node classification benchmark datasets demonstrate the effectiveness of our proposed model.
2 RELATED WORK
Heterogeneous graph neural networks. Heterogeneous graphs have been widely used to solve real-world problems due to a diversity of node types and relationships between nodes. Recently, many HGNNs have been proposed to analyze HINs. HAN(Wang et al. (2019a)) learns node repre- sentations using the node-level attention and the meta-pate petta-pat--eeeel attention. Fu et al. (2020) further consider the intermediate nodes and propose the node-type specific transformation, then use hierar- chical attention structure similar to HAN. Graph Transformer Networks et al. (2019) generates graph by identifying useful connections between unconnected nodes the original graph, then performs graph convolution on the new graph. Zhang et al. (2019) sample heterogeneous neigh- bors for each node by random walk and aggregate node and type level information. Hu et al. (2020) design meta-relation-basee mutual attention to handle graph heterogeneity and implicitly learn meta paths. Nevertheless, these methods require that all nodes have complete and reliable attributes and encounter difficulties in dealing with heterogeneous graphs with the missing and defective attributes.
Learning with missing attributes. Previous handcrafted approaches fill in missing attributes rarely consider graph topology information, making inefficient attributes and compromising model performance. Several deep learning models have been explored. You al. (2020) design a unified framework in which attribute completion is modeled as an edge-level prediction task and the label prediction as a node-level prediction task. Chen al.) (2020) develop a novel GNN framework to perform the link prediction task and the attribute completion task based on distribution matching. Despite tremendous success, they are not suitable for HINS with missing attributes because the com- plex interaction has not been addressed. Recent advance in HGNNs has provided new directions to solve the problem. Jin et al. (202T) attempt to complete missing attributes in a learnable man- ner and propose the framework HGNN-AC, which contains pre-training of topology embedding and attribute completion with attention mechanism. He et al. (2022) design an unsupervised het- erogeneous graph contrastive learning approach for analyzing HINs with missing attributes. Both
2

---

Under review as a conference paper at ICLR 2023
sumpling
augmentation Augmented HINs
Consistenry tmining
The HIN with missing
HGNN
Yu
An example of completing missing attributss
Share params
HGNN @1AX,6)
YIM)
C
oon
An example of calibruting defective attributes
Figure 1: The overview of ARHGA framework. ARHGA performs random sampling (a) to produce ANS. The attributes nodes the ANS are completed or calibrated through attribute aug- mentation (b) to generate multiple graph augmentations (c), which are fed into to shared HGNNs to construct the consistency loss (d). The upper box shows an example of missing attributes sampling and the lower represents example of defective attributes in another sampling.
methods only complete the missing attributes, while our proposed method not only completes the missing attributes but also calibrates defective attributes enhance the quality of attributes.
Graph data augmentation. Recently, variants of data augmentation techniques have been explored in deep graph learning and demonstrated remarkable results. Graph data augmentation aims gen- erate the augmented graph(s) through enriching or changing the information from the original graph. can be categorized into three classes: attribute-wise augmentation, structure- -iie augmentation and label-wise augmentation(Ding et al. (2022)). Among the attribute-wise augmentation strate- gies, there are few lines of existing works for attribute calibration/denoisingg method(Xu et al.] (2022)) is compute the gradient of specific objective functions w.r.t. the node attribute matrix and calibrate the node attributes matrix based on the computed gradient. addi- tion, as special case of noisy setting on graph data, the problem of missing attributes also has been studied. Representative works include GCNMF and Feature Propagation. GCNMF GCNMF(Taguchn et al. (202I) completes the missing data by Gaussian Mixture Modcl(GMM)). It integrates graph representation learning with attribute completion and can be trained in end-to-end manner. Feature Propagation(Rose al. (02T) reconstructs the missing node attributes based diffusion-type differential equation on the graph. It is important to note that the above-mentioned graph augmenta- tion methods only work on homogeneous graphs, heterogeneous attribute-wist augmentation still an under-explores remains problem.
straightforward
METHODOLOGG
In this section, introduce the ARHGA framework, of which the general idea is to tackle the missing and defective attributes simultancously through an effective graph augmentation strategy, and integrate the augmentation process and HGNN module into a unified framework benefit the performance on node classification tasks. Figuee illustrates the proposed framework.
3.1 RANDOM SAMPLING
Random sampling randomly selects part of nodes to form the augmentation node set(ANS) in which nodes may have no attributes or have original attributes. Specifically, given a heterogeneous graph G with node set V/ of n nodes, edge set & and attribute matrix X, randomly sample binary mask E, ~ Bernoulli(1
5) for each node U; V to determine whether v, belongs to the ANS S, where 5 is the probability when E; takes O. We obtain ANS {uile, 0), ie., if e; 0. then 2 E ANS. The attributes of nodes in the ANS are dropped in this step and reconstructed in the phase of attribute augmentation.
3

---

Under review as a conference paper at ICLR 2023
To reconstruct attributes for all nodes, random sampling is performed repeatedly to guarantee that all nodes are selected. the number of samplings is set as M, we give the constraint under which M should satisfy from a probabilistic view. Let P denote the probability of all nodes being selected at least once, since each node is sampled independently, we have:
P P= II ===)(p))
(I)
11=1-P -
(2)
where P; and PC are the probability that node v; is selected at least once and that is not selected respectively, k is the number of times being selected. Given a threshold 7 close enough to 1, when P is greater than T. it is considered having the full coverage of all nodes after M random samplings. Substituting Eq. into [p. can be deduced that M should satisfy (1 8a y > T, i.e., M > log,(1 7-)
Note that the sampling procedure is only performed during training. During inference, we directly consider the node set V as the ANS.
3.2 ATRRIBUTE AUGMENTATION
In heterogeneous graphs, attribute information and structure information are two crucial charac- teristics, and are semantically similar to each other due to the homophily of network et al. (2001): Pei et al. (2020); Scchiccttrull et al. (2018)). With this principle, there is a reason- able hypothesis that the topological relationship between nodes can well reflect the relationship of nodes' attributes. Or, briefly, adjacent nodes tend to have similar attributes. So attributes of nodes in the ANS can be reconstructed based on attribute information of their neighbors. Considering the complex interactions heterogeneous networks, the node topological embeddings H learned to capture underlying structure information, and used as guidance to reconstruct attributes.
Topological embedding, In graph G. the node set associated with corresponding node-type set FF and the edge set & is associated with corresponding edge-type set R. To capture the structure information in G, random walk (Huang et al. (2019) is adopted based on multiple common meta- paths, to generate more comprehensive node sequences which are fed into heterogeneous skip- gram model to learn node topological embeddings. Given a pre-defined meta-path P. : Fi Ri,
F2 R2
Ri-1 Fj, for node vs with type F,, the transition probability at step i can be formulated
as:
p(1441/0, P)= P)- - ( (ott1, ot) E ( = Fi+1 0,
otherwise
where NE., (v:) stands for the neighbors with type F++1 of node e, The random walk makes that underlying semanties information in graph preserved properly. Then skip-gram grikooooo et al. (203)) with one-hot vector as input is adopted to learn topological embeddings by maximizing the probability of the local neighbor structures captured by the random walk, that
max ΣΣ Σ BEV FET (ENF(1)
logp(ulc;0),
where (v) the set of neighbors with type F of node v, which is sampled by random walk based on the given metapath. The topological embedding of nodes can be learned and denoted as H.
Attribute-wise augmentation. From the perspective of network science, the information on direct links is more essential, so one-hop neighbors have more contributions to attribute augmentation. And we notice that directly connected neighbors have different importance because the neighbors of node may be of different types and different local structure in heterogeneous graphs. ARHGA adopts the attention mechanism to learn the importance of different one-hop neighbors based on topologiaal embedding H. Note that ARHGA only computes attention coefficients of nodes' direct neighbors by performing masked attention, which reduces unnecessary computation and is thus more efficient.
Specifically, given a node pair (u, u) which are directly connected, the attention layer learns the importance e,v. which indicates the contribution of node node v, Ntt,, where A&+ denotes
4

---

Under review as a conference paper at ICLR 2023
the one-hop attributed neighbors of node v, the importance can be computed
cvu = Wh.),
(5)
where hr and h are the topological embeddings of node u and node v. Here the attention layer, parametrized by weight matrix W, is shared for all node pairs, and a is an activation function. To better compare the importance across different nodes, the softmax function is used to normalize them to get weighted coefficient Quvi
exp Cou
O'u = softmar(eee EVEN exp(ees)
(6)
Then, ARHGA obtains attributes of node e by aggregating attributes of it's one-hop neighbors ac- cording weighted coefficients C.uu
Σ
Cvu
WEN.
In this way, the missing attributes are completed and original defective attributes are calibrated. As shown in Figure 2 we give brief explanation on the author node(A1) in DBLP and the pa- per node(P1) ACM as an example. The node Al has no attributes and the node P1 has original defective attributes, their attributes reconstructed effectively using proposed framework. Im- portantly, the completed attributes are confirmed and the calibrated attributes are more meaningful than original attributes owing integration with underlying semantic information in heterogeneous graphs after conducting weighted aggregation.
f
A
Papor
hhor --- Aggregation
Paper
Apurogation Ton
(a)
(b)
Figure 2:
(a) An illustration of completing the attributes of node Al DBLP dataset. (b) An illustration of calibrating the attributes of node P1 in ACM dataset.
Finally, to stabilize the learning process and reduce the high variance (brought by the heterogeneity of graphs), we extend the attention mechanism multi-head attention, as done in many existing peethod((Velčkkond et al. (017)) Wang et al. (2019a)). Specifically, K independent attention mech- anisms are performed to serve as the final attributes of node v, and then their outputs are averaged to generate the following attributes of node e:
K
H. = mean
Σ
AvuTa)
(8)
WEA'
where mcan(-) represents average function and K is time of performing attention mechanism. After attribute-wise augmentation, attributes of all nodes are updated as:
X X ANS, ANSS..
3.3 CONSISTENCY TRAINING
After performing random sampling and attribute augmentation for M times, we generate M aug- mented attribute matrices (x(m) < m < M), each of which together with the original graph structure is sent into HINs model to get the corresponding output:
(m) (A,,X* {(m)),
(10)

---

Under review as a conference paper at ICLR 2023
where (()) are the prediction probabilities of (m),,, T denotes HINs model. With the augmented attributes, ARHGA can be applied to any other heterogeneous graph models and successfully en- hance the performance. To be specific, MAGNN(Fu et al. 2020 is combined when implementing ARHGA.
Supervised Loss. Under semi-supervised setting, with labeled nodes among nodes, the pervised loss on the node classification task is defined as the average cross-entropy loss over M augmentations:
Saup
1 M
Σ
8-1 Y log y,(m)
(11)
where {0, 1 j are ground-truth labels with C' representing the number of classes.
Consistency Loss. Valid graph data augmentation changes the input in way that has relatively trivial impact on the final node classification. We naturally embed this knowledge into our model by designing consistency loss over M augmentations.
For each node vú, we generate "guessed label" no matter it originally has a label or not. Specifically, find the average of the model's prediction distributions across all the M augmentations of ey:
Y,
Σ
M
(m)
(12)
M
m
then the "guessed label" Y = (%i,.... Yi,, Pio-1) for node v; is computed through the sharpening trick(Berthelot al. (2019)), in which Y, is the guessed probability of u; belonging to class j:
C-1
8/ = 17 Σ -SiSCC,,,
(13)
where < T S acts as the "temperature" that controls the sharpness of the distribution. In ARHGA, T is set as a small value to enforce the "guessed label" to approach one-hot distribution. Then minimize the distance between Y, and Y to design the consistency loss:
Loon
I Y 11 1
(14)
M
1=0
Optimization Objective. Both the supervised loss and the consistency loss are combined as the final loss of ARHGA:
L= Loup + XLcons
(15)
where the papee-parrmeter/ À is introduced to control the balance between the two losses. By min- imizing the final loss, our model can be optimized via back propagation in an end-to-end manner. Algorithm (present in Appendix A.1) outlines ARHGA's training process. During inference, we di- rectly take V as the ANS, that is, attributes of all nodes are reconstructed after performing attribute augmentation one time. Hence, the inference formula Y = ((A, X, O), where X (X₂ (X,|;; E V} and Y is the corresponding prediction probabilitics.
EXPERIMENTS
In this section, we first give the experimental setup, and then evaluate the performance of ARHGA on the node classification and report the visualization results. We also conduct a deep analysis on the effectiveness of ARHGA. Finally, we investigate the sensitivity with respect to hyper-parameters of ARHGA.
Datasets. We conduct experiments on three widely-used HINs datasets, i.c., DBLF ACM IMDE to analyze the effectiveness of ARHGA. Note that, only paper nodes in DBLP and ACM,
https://dblp.uui-ttree..e hhtpss//dl..cm.oom http://www.mmdh.com

---

Under review as a conference paper at ICLR 2023
Table 1: Results(%) of node classification three datasets
Datasets
Metrics
Training
metaputh2voc
GCN
HerGNN
HAN
MAGNN
HGNN-AC
ARHGA
10%
91.09
89.53
64.57
91.09
91.24
92.44
93,80
93.86
20%
91.50
90.06
66.92
91.72
91.69
92.33
93,92
94.12
Macro-F1
40%
92.55
90.37
73,23
91.84
92.97
94.06
94,44
60%
93.25
90.57
77.17
92.26
92.01
93,30
94.04
94.52
93.77
94.22
DBLP
10%
91.74
90.02
75.90
91.64
91.88
93.02
94,22
94.31
20%
92.14
90.53
76.98
92.23
92.30
93.08
94.34
94.55
Miemo-FI
40%
93.09
90.83
79.61
92.55
92.46
93.50
94.46
94.84
60%
93,76
91.01
81.62
92.79
92.65
93.83
94,46
94,91
80%
93,94
82.22
94.27
94.61
95.17
10%
67.81
70.79
89.29
88,20
89.39
86,60
90.29
92.007
20%
69.95
70.41
89.59
89.16
90.01
$8.01
91.51
92.94
Macro-FI
40%
71.15
70.82
89.77
90.14
90.82
89.42
92.75
94,02
60%
71.74
69.67
89.72
90.71
91.51
90.39
93.46
94_36
80%
72.18
67.23
89,42
91.71
90.79
93.79
94.57
ACM
10%
10.29
74.10
89.19
88,18
89.32
86.67
90.443
91.95
89.47
92.82
Micro-F1
40%
73.17
74.57
89.65
90.11
90.73
89.48
92.9
93.94
60%
73.65
74.10
89.60
90.64
91.37
90.42
93.57
94.29
80%
74.14
72.69
89.29
90.93
91.56
90.80
93.87
94.52
10%
44,29
43,70
33.61
45.68
$7.02
56.39
58.69
59.39
58.11
60.45
Macro-FI
40%
47.70
45.26
$5.09
$1.61
$7.75
$9.39
60.18
641.98
48.25
47.70
55.71
59.97
60.60
61.39
80%
48.73
48.25
55.40
53.24
$7.23
60.02
60.75
61.71
10%
46.15
47.02
54.14
46.56
$7.35
56.33
58.97
59.60
20%
48.08
47.44
55.02
$7.82
58,16
59.84
60.53
Micro-F1
40%
49.55
47.62
55.29
52.47
$7.98
59.46
60.38
61.09
60.79
61.52
80%
50.68
48.73
55.67
54.25
57.46
60.15
60.98
61.88
and movie nodes in IMDB have original attributes while other types of nodes have on attributes. More details present in Appendix A.2.1.
Baselines. We compare ARHGA with seven state-of-the-an methods representative two traditional homogeneous GNNs, i.e., GCN(KKif & Welling (206)).
different categories:
CAT/Veličkove et al. (2017)), and five heterogeneous graph models, i.e., (eetaptt2vve(DDnn et al.
(2017). HetGNN(Zhang
al. (2019)) HAN(Wang al. (209a)), MAGNN(Fu et al. (2020).
MAGNN-AC(in et al. (021)). Among heterogeneous graph models, metapath2vec is network embedding methods and other models are GNN-based methods. MAGNN-AC an approach to complete missing attributes of nodes having attributes in heterogenerous graphs.
Parameter Settings. For the settings/parameters of different baselines, we use the default hyper- parameters suggested in MAGNN-AC. The embedding dimensions of all methods is set to 64 for fair comparison. ARHGA, random sampling probability mentations M to 6. The number of attention heads K is set to 8 and À is 0.5 for the weight of the consistency loss. In addition, ARHGA is trained by adopting Adam optimizer(Kingmm (2014)) with the learning rate 0.005. The early stopping strategy with patience of 5 epoches is adopted on validation set our experiments.
to 0.5 and the number of aug-
Ba
4.1 NODE CLASSIFICATION
In this section, the performance on node classification of ARHGA is compared with that of different models. First, we generate embeddings of labeled nodes(i.e., authors DBLP, authors in ACM and movies in IMDB), and then feed them into a linear support vector machine(SVMI) ( classifier with different training ratios from 10% to 80%. Table [ reports the results averaged Macro-F1 and Micro-Fl over 5 times, where the best results in bold fonts.
(200T))
As shown, ARHGA consistently achieves a significant margin over baselines and datasets. After conducting attribute augmentation, ARHGA has 0.9%-5.47%, higher accuracy than MAGNN and 0.06% higher than MAGNN-AC, another MAGNN MAGeN-baasd method, which performs the best results from the baseline methods. Specially, although accuracy on DBLP dataset, ARHGA still has nontrivial improvement. The improvement is primarily due to the fact ARHGA provides better representation of node attributes. In addition, GNN-based
methods have already achieved high

---

Under review as a conference paper at ICLR 2023
heterogeneous methods perform better than metapath2vec since attributes information is integrated, demonstrating the importance node attributes. The poor performance GCN and GAT reveals the importance of encoding heterogeneous semantic information when analyzing HINs. Our proposed method not only captures and preserves underlying semantic information in heterogeneous graphs but also effectively reconstructs node attributes augment graph, which contribute the great superiority of model.
4.2 VISUALIZATION
For more intuitive comparison, we conduct the task of visualization by learning embeddings of paper nodes of MAGNN, MAGNN-AC and ARHGA on the ACM dataset. Then the well-known Van der Maaten & Hinton (2008) is utilized to project the embeddings into 2-dimensional space, where nodes with different colors belong to different class.
(a) MAGNN
MAGNN-AC
(c) ARHGA
Figure 3: Visualization of embeddings of paper nodes in ACM. Different colors correspond dif- ferent research ground truth.
As shown in Figure with the reconstructed attributes, ARHGA has the clearest boundary and densest cluster structure to classify nodes among the three methods. contrast, MAGNN and MAGNN-AC perform poorly, some paper nodes of different classes are mixed and overlapped ob- viously. MAGNN adopts imputation methods to fill in the missing attributes, which provides little useful information for node classification, resulting in poor performance. MAGNN-AC only com- pletes the missing attributes and the defective attributes are not addressed properly. ARHGA not only completes missing attributes but also enhances the quality of attributes to make nodes more distinguishable.
4.3
A DEEP ANALYSIS OF ARHGA
WARIGN
AARRICA
065
090
11.0
200
005
170
M0%
605
80%
(a) DBLP
(b) ACM
(c) IMDB
Figure 4: Comparisons of ARHGA with two variants(ARHGA-L. ARHGA-2) and MAGNN on node classification.
To verify the effectiveness of attribute completion and attribute calibration in our method, we con- duct experiments comparing ARHGA with two variants. The variants given follows: 1) ARHGA of removing attributes calibration, only completing the missing attributes, named ARHGA- 2) ARHGA of removing attributes completion, only enhancing the quantity of the original defec-
8

---

Under review as a conference paper at ICLR 2023
tive attributes, named ARHGA-2, in which, the missing attributes are obtained by imputation strate- gies. In addition, we also compare with MAGNN to better analyze the benefit of ARHGA. Figure presents the results of the deep analysis. More results see Appendix A.2.2.
As shown, ARHGA has the best results on all datasets at all label rates. Though ARHGA-1 and ARHGA-2 lower than ARHGA, still outperform MAGNN. This is mainly because ARHGA-1 completes missing attributes accurately and ARHGA-2 calibrates the defective attributes effectively, while there are no corresponding strategies to tackle missing and defective attributes in MAGNN. Furthermore, all the results of ARHGA-1 are better than those of ARHGA-2, so it is believed that attribute completion contributes more than attribute calibration The overall results show that accurate attribute information palys significant role in learning node representations. And ARHGA is an effective exploration of attribute reconstruction through graph data augmentation.
the effectiveness of ARHGA.
4.4 PARAMETER ANALYSIS
In this section, we investigate the sensitivity of ARHGA to the critical sampling probability number of augmentations M and weight of consistency loss X. We report the results of node classification by varying these parameters on ACM dataset.
random
Marro
0040
Mim-71
0.33
0:00
0:00
(a) The parameters M and 5
(b) weighted coefficient À
Figure 5: Sensitivity analysis on ACM dataset. We report the average result of the node classification
across different training ratios.
From Figure (a) the performance of ARHGA fluctuates slightly when 5 and M are in certain ranges, and drops obviously when 8 and M are set too small. This is because small d and M may result in attributes of some nodes not being reconstructed, impairing the effectiveness of graph augmentation. From Figure 5(b) we observe that the performance of ARHGA shows a trend of first rising and then slowly decreasing when X takes different values from 0.2 to 0.8, and achieves the best when X 0.5. This observation indicates that the consistency loss and supervised loss are both important, each of them contributes to the training of ARHGA. The overall results suggest that ARHGA is relatively stable for hyperparameter perturbation, demonstrating the insensitivity of ARHGA. The results on DBLP and IMDB dataset(provided in Appendix A.2.3) also indicate the similar sensitivity.
5 CONCLUSIONS
In this paper, we present a novel generic architecture(ARHGA) to reconstruct attributes in hetero- geneous graphs. In ARHGA, we design an effective attribute augmentation strategy, which not only solves the problem of missing attributes, but also enhances the quality of original defective attributes. The augmentation strategy is essentially to conduct weighted aggregation through the attention mechanism guided by the topological relationship between nodes. The results of node classification show its consistent performance superiority over seven state-of-the-art baselines on benchmark datasets. The deep analysis of ARHGA demonstrates the effectiveness of completion of missing attributes and calibration of defective attributes. We conclude that ARHGA provides better attribute representation helpful for improving semi-supervised classification heteroge- neous networks. In future work, we aim to design a structure- wise augmentation for a better graph structure representation in heterogeneous graphs.
augmentati

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 0rXiv:1409.047.. 2014.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural informa- fion processing systems, 32, 2019.
Xu Chen, Siheng Chen, Jiangchao Yao, Huangjie Zheng, Ya Zhang, and Ivor W Tsang. Learning on attribute-missing graphs. IEEE transactions on pattern analysis and machine intelligence, 2020.
Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning:
A survey, arXiv preprint arXiv:2202.08233, 2022.
Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 135-144, 2017.
Tao-yang Fu, Wang-Chier Lee, and Zhen Lei. Hin2vec: Explore meta-paths in heterogeneous information networks for representation learning. In Proceedings the 2017 ACM Conference on Information and Knowledge Management, pp. 1797-1806, 2017.
Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference 2020, pp. 2331-2341, 2020.
Dongxiao He, Chundong Liang, Cuiying Huo, Zhiyong Feng, Di Jin, Liang Yang. and Weixiong Zhang. Analyzing heterogeneous networks with missing attributes by unsupervised contrastive learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of The Web Conference 2020, pp. 2704-2710, 2020.
Xiao Huang, Qingquan Song, Yuening Li, and Xia Hu. Graph recurrent networks with attributed random walks. In Proceedings the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 732-740, 2019.
Di Jin, Cuiying Huo, Chundong Liang, and Liang Yang. Heterogeneous graph neural network via attribute completion. In Proceedings of the Web Conference 2021, pp. 391-400, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv:1412.668 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609000609 2016.
Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In International confer- ence on machine learning, pp. 3734-3743. PMLR, 2019.
Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of a feather: Homophily in social networks. Annual review of sociology, pp. 415-444, 2001.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.33 2013.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. arXiv preprint urXiv:2002.05287 2020.
Emanuele Rossi, Henry Kenlay, Maria I Gorinova, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael Bronstein. On the unreasonable effectiveness of feature propagation learning on graphs with missing node features. arXiv preprint arXiv:21/1. 2021.
10

---

Under review as a conference paper at ICLR 2023
Michael Schlichtkrull, Thomas N Kipf. Peter Bloem, Rianne van den Berg. Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593-607. Springer, 2018.
Jingbo Shang, Meng Qu, Jialu Liu, Lance M Kaplan, Jiawei Han, and Jian Peng. Meta-path guided embedding for similarity search in large-scale heterogeneous information networks. arXiv 2016.
preprint
Wei Shen, Jiawei Han, Jianyong Wang. Xiaojie Yuan, and Zhenglu Yang. Shine+: general frame- work for domain-specific entity linking with heterogeneous information networks. IEEE Trans- actions on Knowledge and Data Engineering, 30(2):353-366, 2017.
Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and S Yu Philip. A survey of heterogeneous information network analysis. IEEE Transactions on Knowledge and Data Engineering, 29(1): 17-37, 2016.
Yizhou Sun and Jiawei Han. Mining heterogeneous information networks: a structural analysis approach. Acm Sigkdd Explorations Newsletter, 14(2):20-25 2013.
Johan AK Suykens. Support vector machines: nonlinear modelling and control perspective. Eu- ropean journal of control, 7(2-3):311-327, 2001.
Hibiki Taguchi, Xin Liu, and Tsuyoshi Murata. Graph convolutional networks for graphs containing missing features. Future Generation Computer Systems, 117:155-168, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 10903, 2017.
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip Yu. Heterogeneous graph attention network. In The world wide web conference, pp. 2022-2032 2019a.
Xiao Wang, Yiding Zhang, and Chuan Shi. Hyperbolic heterogeneous information network embed- ding. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5337-5344, 2019b.
Qizhe Xie. Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33:6256-6268. 2020.
Zhe Xu, Boxin Du and Hanghang Tong. Graph sanitation with application to node classification. In Proceedings of the ACM Web Conference 2022, pp. 1136-1147, 2022.
Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. Heterogeneous network represen- tation learning: Survey, benchmark, evaluation, and beyond. 2020.
Jiaxuan You, Xiaobai Ma, Yi Ding, Mykel J Kochenderfer, and Jure Leskovec. Handling missing data with graph representation learning. Advances in Neural Information Processing Systems, 33: 19087, 2020.
Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph trans- former networks. Advances in neural information processing systems, 32, 2019.
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heteroge- neous graph neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 793-803, 2019.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pp. 2069-2080, 2021.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PSEUDOCODE OF ARHGA
Algorithm ARHGA
The node set V, adjacency matrix attribute matrix X E Raxd time of augmentations M in each epoch, the number of attention head K, learning rate a HIN (o,el:
Input:
Output: Prediction Y.
1: while not convergence do
2: for m = M do
3: Randomly select nodes from V to compose the ANg(m)
K
4: Perform attribute augmentation on the AN ... = mean
Σ Σ avata)
k NEA
5: Predict probability distribution using the HINs model: ((() = ( (m)) Q)
end for
M
Compute the supervised loss Loup
Σ
Σ 1=0
Y
log p((m)
M
and the consistency loss Loon 37
Σ
72
Σ
|
r, (m) 12
m=1
8: Update the parameters by gradients descending: end while
θ θ 17 ( XLcon)
9:
10: Output prediction Y via: y = (AA,8,0)
A.2 MORE EXPERIMENTAL RESULTS
In this section, we provide dataset details and more experimental results besides the results in the main paper.
A.2.1
ADDITIONAL. DATASET DETAILS
In this section, we provide some additional, relevant dataset details, the statistics of these datasets are shown in Table AT
Table Al: Statistics of the datasets
Datasets
Nodes
Edges
HasAttrbbues
Author(A):4057
A-P:19645
Paper(P):143288
DBLP
P-T:85810
Paper
Term(T):7728
P-V:14328
Venue(V):20
Paper(P):4019
P-P:9615
ACM
Author(A))7167
P-A:13407
Paper
Subject(S):60 P-S:4019
Movic(M):4278
M-D:4278
IMDB
Director(D):2081
Movie
M-A:12828
Actor(A)::257
DBLP: This is a subset of DBLP with 4057 authors(A), 14328 papers(P), 20 venue(V), and 8789 terms(t). The authors are divided into four research areas: database, data mining, machinesearch- ing, and information retrieval. Only paper nodes have attributes derived from their keywords, and other nodes have no raw attributes.
ACM: This is a subset of DBLP with 4057 authors(A). 14328 papers(P), 20 venue(V), and 8789 terms(t). The authors divided into four research areas: database, data mining, machinesearch-
12

---

Under review as a conference paper at ICLR 2023
ing, and information retrieval. Only paper nodes have attributes derived from their keywords, and other nodes have no original attributes.
IMDB: We extract subset of IMDB with 4278 movies(M), 2081 directors(D), and 5257 tors(A). The movie nodes are labeled by their genres: action, comedy and drama. In this dataset, movies described by bag-of-words representation of their plot keywords, and other nodes have no original attributes.
A.2.2 A DEEP ANALYSIS OF ARHGA
We also report the Micro-F1 values on three benchmark datasets evaluate the effectiveness of attribute completion and attribute calibration in ARHGA. Figure ( presents the results, which is similar that of ACM dataset.
-34VEN
WARRIS ARETA
+MAGN AABRAA WAREENS MARIOO
+396220 YARMON
WARNAN- RARMIA
100
2010
100
(a) DBLP
(b) ACM
IMDB
Figure 6: Comparisons of ARHGA with two variants ARHGAA--- and MAGNN on node classification.
A.2.3 PARAMETER ANALYSIS
We investigate the sensitivity of critical Fl values in Figure 7 and Figure g hyperparameter perturbation.
on DBLP and IMDB and report Macro- The results also demonstrate the insensitivity of ARHGA to
0.23
0481
080
640
698
000
6m
(a) DBLP
(b) IMDB
Figure 7: Analysis of the parameter sensitivity of A. We report the average result of the node classification across different training ratios.
(a) DBLP
(b) IMDB
Figure 8: Analysis of the parameter sensitivity of M and 5. We report the average result of the node classification across different training ratios.
13


Reviews:
[{"review_id": "OQT8pHuh9bI", "reviewer": "Reviewer_Aba8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed ARHGA, a heterogeneous graph augmentation method for attribute reconstruction. It solved both missing attributes and defective attributes problems in heterogeneous information networks. Extensive experiments prove the effectiveness of proposed attribute reconstruction framework.", "summary_of_the_review": "This paper provides a graph augmentation method to solve the attribute reconstruction problem in HINs, however the contribution is incremental and the experiments could not fully support all the contributions.", "strength_and_weaknesses": "Strength:\n\n-- The attribute reconstruction problem in heterogeneous information networks is important. The topic of this paper is relative.\n\n-- The proposed method is effective in some extent, which is proved by many experiments.\n\n-- The method is introduced clearly and the paper is easy to follow.\n\nWeakness:\n\n-- The main contribution is a little incremental. In one hand, the framework is based on existing techniques. In the other hand, this paper does not provide any novel theoretical insights on this issue. \n\n-- In my opinion, the experiments are not quite sufficient in following two perspectives. 1) The defective attributes are not covered in the experiments. One of the main contributions is to jointly tackle missing and defective attributes. The adopted datasets naturally have missing attributes, but there are no evidences that the attributes of these datasets are defective. It's better to change the original attributes randomly or exploit some simple adversarial attacks on attributes to make the original attributes defective. 2) There could be more downstream tasks other than node classification. Attribute reconstruction is widely used in many scenarios e.g. anomaly detection, unsupervised or self supervised learning. It's better to show that the proposed ARHGA framework could support different downstream tasks to prove the effectiveness of ARHGA.\n\n-- Some typos and small writing problems, e.g. the mapping $\\phi$ in Equation (3) is not explained in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MtFXCZpDQOM", "reviewer": "Reviewer_Qaru", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a method, ARHGA, for inferring missing attributes and calibrating defective attributes in a heterogenous graph. The main idea of the paper is to subsample the graph and augment the attributes. The attributes were filled in using topological embeddings (based on random walk 2 vec on meta-paths) and a one-hope attention mechanism. \n\n", "summary_of_the_review": "I recommend rejection due to a lack of novelty as explained above. ", "strength_and_weaknesses": "The idea of dealing with missing/corrupted attributes in heterogeneous graphs is appealing. The authors proposed a method to tackle the problem by leveraging techniques like subgraph random sampling, graph embedding/attentions, and dual losses (supervised/consistency) is sound. \n\nHowever, I do notice a similar method, HGNN-AC (WWW 21'). HGNN-AC is also cited in this work but not really thoroughly compared and discussed. Moreover, there is another similar method, (He et al., 2022). Both methods were only briefly mentioned in section 2. The only key difference that the author pointed out is they only complete missing attributes but do not fix defective attributes. Notably, both HGNN-AC and He et al., use almost an identical approach: random drop attributes, use topology embedding + attention, and dual losses (prediction loss + completion loss). The experiments also identically follow. \n\n1) The authors need to better explain the novelty of this work and how it compares to HGNN-AC. \n2) For experiments, only node classification was shown. Are there other possible tasks we can try? \n3) The experiments are solely based on dropping attributes, how about attributes corruption/defective attributes? \n4) How are \"defective attributes\" defined? Are there side effects of trying to update the attributes?\n5) Could we also do a comparison with He et al., 2022 work? I understand it is unsupervised but the results in table 3 from He's work looks similar to ARHGA. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "B28ZhTx_T5", "reviewer": "Reviewer_FtXi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a joint learning on node classification and attribute augmentation on heterogeneous graphs. The attribute augmentation is based on the aggregations among one-hop neighbors weighted by attentions. Experiments show the proposed approach results in marginal improvements over the baseline methods.", "summary_of_the_review": "Overall, the paper has limited novelty and contributions to be accepted:\n- It's not clear how to set M based on delta. When delta is close to 1, M can be very large, which will be inefficient.\n- The key idea of attention-based aggregation among one-hop neighbors (e.g., Eq. 7) is not meaningful. Different types of nodes have different input node attributes, thus directly aggregating x_u where u can be of any types does not make any sense. E.g., aggregating x_u where u is a paper for a venue node.\n- The experimental results in Table 1 only shows very limited improvements over the baseline methods. \n- Results are not reported by the mean and standard deviations.", "strength_and_weaknesses": "Strengths:\n- The problem of heterogeneous graph augmentation is interesting and important.\n- Experimental results can show some improvements.\n\nWeaknesses:\n- For Eq. (2), the authors claim that delta should be close enough to 1, but in the experiments, it's set to be 0.5 only\n- The attention mechanism used in the paper is problematic. Attention-based aggregation is designed and applied to be among all types of nodes for attribute augmentation.\n- The experimental results in Table 1 only shows very limited improvements over the baseline methods. \n- Results are not reported by the mean and standard deviations.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject