Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose to leverage structural clustering for pre-training heterogeneous graph neural networks in a self-supervised manner. The proposed SHGP framework adopts Attentive HGNN as the base encoder and utilizes label propagation for structural clustering, thereby deriving pseudo labels for  self-supervised pre-training. The experiments are conducted on several benchmark datasets of heterogeneous graphs. The experimental results show that the proposed SHGP outperforms several conventional baseline methods in both tasks of object classification and clustering.
strengths_and_weaknesses: Strengths
* Simple but effective framework.
* Publicly available benchmark datasets and released implementations for the reproducibility. 


Weaknesses
* The idea of using clustering and pooling to benefit graph neural nets is not novel. 
* Do not propose novel methods directly related to heterogeneous graphs.
* Some state-of-the-art pertaining methods for both general and heterogeneous graphs [a, b], as well as some unsupervised representation learning methods [c], are not discussed and compared in the experiments.

    * [a] Chien, E., Chang, W. C., Hsieh, C. J., Yu, H. F., Zhang, J., Milenkovic, O., & Dhillon, I. S. (2021, September). Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction. In International Conference on Learning Representations.
    * [b] Wang, X., Liu, N., Han, H., & Shi, C. (2021, August). Self-supervised heterogeneous graph neural network with co-contrastive learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 1726-1736).
    * [c] Jiang, J. Y., Li, Z., Ju, C. J. T., & Wang, W. (2020, October). Maru: Meta-context aware random walks for heterogeneous network representation learning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 575-584).

* No theoretical supports.


questions: * The concept of considering clustering for graph neural networks is very similar to the concept of pooling. For example, [c,d] also learn a soft cluster assignment for enhancing model learning. The authors should compare with them and emphasize the motivation of the paper.

    * [c] Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., & Leskovec, J. (2018). Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31.
    * [d] Ranjan, E., Sanyal, S., & Talukdar, P. (2020, April). Asap: Adaptive structure aware pooling for learning hierarchical graph representations. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 5470-5477).

* The whole framework combines two established components from other studies, but the key idea is still about structural clustering that is unnecessarily relevant to heterogeneous graphs. I wonder if the idea can be adopted with models for homogeneous graphs.

*Some state-of-the-art pertaining methods for both general and heterogeneous graphs [a, b], as well as some unsupervised representation learning methods [c], are not discussed and compared in the experiments. It would be better to compare with the state-of-the art methods in the experiments.
limitations: The authors do not mention any limitations and potential negative societal impact of the work. However, graph neural net related algorithms could still have some risks about resulting some abuse scenarios on social networks.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the problem of learning object embeddings in heterogeneous information networks. To address this problem, the paper proposes a self-supervised graph learning model SHGP, which consists of two modules, namely Att-HGNN module and Att-LPA module. The main idea of the proposed method is to use the Att-LPA module to generate pseudo labels for objects, which are then leveraged to guide the Att-HGNN module to learn object embeddings. Experimental results on object classification and clustering verify the effectiveness of the method. 
strengths_and_weaknesses: The idea of using structural clustering (label propagation method) to generate pseudo labels as self-supervision signals is novel and the experimental results verify the effectiveness of such a method. The paper is clear and easy to understand. 

However, there are some weaknesses in the paper:

(1) Some technical details of the proposed method are unclear:

a) In section 4.1, the paper writes “LPA randomly associates each object with a unique integer as its initial label”. But where do these “unique integers” come from? Are they chosen from the ground-truth label set?

b) How to determine the number of clusters $K$?

c) In line 171, what is the link weight and how to obtain these weights? Similarly, how to obtain the attention coefficients? Are they model parameters?

(2) The motivation of using LPA to generate pseudo labels is not clear. Why not use another HGNN to generate pseudo labels instead?

(3) In the object classification experiment, I don’t think it is fair to use the object embeddings learned by semi-supervised baselines for comparison as they are not designed to learn object embeddings. Instead, I think the paper should compare the results of unsupervised methods with the results obtained by the classifiers of semi-supervised methods. 

Moreover, I am curious about how the number of clusters $K$ affects the performance of SHGP. 

Furthermore, the paper claims that the Att-HGNN module can use any attention-based HGNN encoders. However, the performance of the proposed method with different encoders is unknown. 

(4) The writing of the paper can be further improved:

a) I doubt whether the sentence in line 21 “This success, however, comes at the cost of a heavy reliance on high-quality supervision labels” is correct since most graph neural network models, such as GCN, GraphSAGE and GAT, are proposed to address the semi-supervised classification problem, which does not require a lot of labels. 

b) what do positive examples and negative examples mean? 

c) The full name of the proposed model SHGP is missing. 

d) I doubt whether the sentence “SHGP is the first attempt to perform SSL on HINs without any positive or negative examples” is correct since the paper does not explicitly explain what is positive examples and negative examples. Can we treat the ground-truth edges and object features as positive examples? If so, then the proposed model still uses positive examples. 

e) The citations of datasets are absent in the main paper. 

f) minors: 
In line 99, conceptions —> concepts; 
In line 235, punctuation is missing. 

questions: (1) How to choose the initial labels of objects in LPA module? Please see “strength and weakness” for more details about this question. 
(2) How to choose the number of clusters $K$ and how does this hyperparameter affects the performance of the proposed method?
(3) Why use LPA to generate pseudo labels instead of other models such as HGNN?
limitations: The authors do not discuss the limitations and potential negative social impact of the proposed method in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, a self-supervised heterogeneous pre-training model (SHGP) is proposed. The Att-LPA module obtains node pseudo-labels through structured clustering, and the Att-HGNN module learns node embedding through attention mechanism. SHGP achieved significant results when compared with unsupervised or semi-supervised baseline methods.
strengths_and_weaknesses: Strengths:
1. In this paper, a new paradigm of self-supervised pre-training model on heterogeneous graphs is proposed, which generates pseudo-labels of nodes to guide the training of node embedding.
2. In this paper, pseudo-labels are obtained through label propagation, avoiding the generation of positive or negative examples, thus improving the generalization of the model.

Weaknesses:
1. Pseudo-labels initialized by structural clustering only consider structural features and ignore important information in node features.
2. The randomness of pseudo labels generated by LPA algorithm is large.

questions: 1. Is it necessary to use attention in both ATT-LPA and ATT-HGNN? Ablation study are required.
2. How to overcome the error caused by randomness of LPA algorithm.
3. In Line 5 of algorithm 1, will it be difficult for algorithm to converge if pseudo-labels are continuously updated?

limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies self-supervised learning (SSL) techniques for heterogeneous graph neural networks (HGNN). The proposed SSL method, SHGP, optimized attention-based HGNN layer (Attn-HGNN) by cross-entropy loss between encoder’s predictions and node’s pseudo cluster label obtained from the label propagation algorithm (Attn-LPA). In addition, SHGP demonstrated that iteratively training model parameters based on the updated pseudo labels from Attn-LPA can further improve the model performance.  
strengths_and_weaknesses: ## Strength
- The paper writing is clear and easy to follow
- The empirical performance seems promising

## Weakness
- SHGP introduces several hyper-parameters, including the number of iterations for warm-up (line 2 in Algorithm 1), the number of iterations for main update (line 3 of Algorithm 1), the number of cluster size K.
- Lack of comprehensive ablation study on structural clustering. See Questions section  for further comments. 
- Lack of proper citation for SSL based on dynamically-updated pseudo labels in the literature. For example, HuBERT (Hsu et al.) also considers pre-training tasks based on  pseudo labels, where k-mean clusters are dynamically updated.

questions: 1. It is unclear which part of the structural clustering leads to the final good performance, as there are several hyper-parameters in pseudo node label cluster generation. For example,
- How performance would change if we freeze (or only update only very few times) the pseudo node label, but just optimize the attn-HGCN layer?
- How does the number of cluster sizes (K) affect the embedding quality? Can we also dynamically adjust the cluster size based on some coarse-to-fine heuristic? 
- If each node comes with side information (e.g., text description), can we cluster nodes based on label text representation?
How does the pseudo label assignment change between two attn-LPA updates? For example, how many nodes change their pseudo label cluster?

2. Some implementation and model details are missing in Sec 5.3 and Table 1. For example,
- Comparison model parameters between SHGP and baselines, which showed if the performance gain is due to larger encoder size and node embedding dimensions.
- The pre-training time of SHGP and the baselines.

limitations: No potential negative societal impact.
review_title: null

Meta-review: metareview: I recommend to accept this paper. 

In this paper, the authors propose SHGP, a technique to leverage structural clustering for pre-training heterogeneous graph neural networks in a self-supervised manner. After the rebuttal, all the reviewers are positive to accept this paper. I would encourage the authors to revise the paper based on the suggestions from reviewers. 
confidence: Certain
Decision: Accept