Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Cold Brew, a new method for learning cold start node embeddings in graphs. Cold Brew leverages teacher-student framework (knowledge distillation) to handle nodes without neighbors by transferring knowledge of teacher network (learned from head nodes) to student network (for tail or isolated nodes). Experiments are conducted to show that the proposed method outperforms some baseline methods. 
main_review: Strength 

1 - Propose a knowledge-distillation based technique for learning cold start node representation.  

2 - The problem is relatively new and interesting. 

3 - Presentation is overall good.

Weakness

1 - The novelty of proposed model is not significant. 

2 - Experiments could be improved.    

3 - Lacks related work discussion.

Detailed Review

It is interesting to develop new method for GNN to handle nodes without neighbors. The proposed knowledge distillation framework seems reasonable for me. Besides, FCR metric is proposed to measure the importance of node feature and graph structure. The proposed method works well for node classification task in several datasets. Following are some issues.

The novelty of proposed method is not significant as it follows general teacher-student network and combines both neighbor aggregator and structure embedding to learn node embeddings. These parts are borrowed from existing techniques. It would be better to discuss model contribution. In addition, the current manuscript only studies node classification task while it is also possible to study link prediction task over cold-start nodes (e.g., tail nodes). Moreover, there are some existing works studying tail node representation learning that should be discussed or compared, such as:

Towards locality-aware meta-learning of tail node embeddings on networks, CIKM'21

Tail-GNN: Tail-Node Graph Neural Networks, KDD'21

Minor issues exist, such as typo. For example, the first sentence of section 3.2 should be: to integrate the knowledge of GNN teacher.

--
Update after rebuttal: The authors addressed some of my concerns in experiments. The novelty is still incremental for me. I change my score to borderline above. 
summary_of_the_review: This work studies an interesting problem. The proposed method is reasonable for solving the problem. The novelty is not significant. In addition, experiments could be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Many real-world graphs have power-law distributions of node degrees and learning the representations of nodes with few or even no connections may only depend on their attributes. This paper studies the problem of learning good representations of such nodes using inductive GNNs. It proposes a new method to generalize GNNs better for tail nodes compared to pointwise and graph-based
models using a distillation approach. A metric, feature contribution ratio, has been proposed in quantifying the contribution of nodes' features in predicting labels. Experiments on several graph datasets demonstrate the effectiveness of the proposed method especially in learning better representations of the tail and isolated nodes.
main_review: Overall, this paper is well-organized. Paying more attention to tail nodes and/or nodes with fewer neighbors is important and has been neglected in a lot of previous studies on GNN. It is also of practical value, e.g., cold start in the recommendation system. Using knowledge distill to learn a transformation from both structures and attributes to attribute is an interesting idea to solve this problem. Experimental results on several graphs demonstrate the effectiveness of the proposed method.

My major concerns are as follows:
- There are some previous studies on tail node representation learning, e.g. [1] and [2]. I suggest that the authors discuss these studies and compare the experimental result with these methods.
- What's the reason for minimizing the structural embedding E in the loss function (Eq (3))? Since E represents the node-wise representation with label information, it is not intuitively clear why this embedding should be as small as possible.
- The experimental studies only validate the performance using GCN. I wonder if the performance will be influenced by different GNN. For example, GraphSage uses a sampling strategy to aggregate embedding, and intuitively this may mitigate the impact of imbalanced distribution or noisy structural information.
- Since label information has been incorporated into the embedding (with the structural embedding E), the ratio of labeled nodes may influence the performance. Could you give some empirical or theoretical analysis of this possible relationship between the performance and ratio of labels?

I also have some minor comments:
- Title: The title contains the terms incomplete or missing, but the main content discusses a more general case including tail nodes, isolated nodes, and nodes with (maybe) incomplete or missing neighborhoods. Please make the title and content consistent.
- Notation z has been used as some element under Eq (4) and performance of models in Eq (5).
- Typo on Page 4: this *motivate* us to strengthen..
- The information in Figure 1 top is clear but the way to show the graph (the grey nodes) on such a coordinate is misleading.

[1] Tail-GNN: Tail-Node Graph Neural Networks, KDD 2021
[2] Towards Locality-Aware Meta-Learning of Tail Node Embeddings on Networks, CIKM 2020
summary_of_the_review: This paper studies an interesting and practical problem of learning better representations for tail and isolated nodes. Making use of a distilling approach, the combined information of both structures and attributes can be learned to generalize in nodes with less or even no structural information. Some weakness of the paper includes missing baselines, comparison to other GNNs, and deeper investigation of the label information.
-------------------------------
I appreciate the responses from the authors that addressed most of my concerns. I updated my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work focuses on a very practical problem of strict cold start(SCS) recommendations which is a highly prevalent and relevant problem. The author’s main contribution to address the SCS is to use GNN with knowledge distillation – this proposed solution does not have to rely exclusively on the node features. The authors also define an FCR (feature contribution ratio) that can help determine the ideal network architecture – FCR can optimize the model selection which significantly affects the quality of overall system performance.

main_review: The proposed method to adopt GNN with knowledge distillation to solve SCS problem relies on well-known concepts and previous works, but the learnable Structural Embedding and model selection methodology is novel in contribution and can also have practical adoption for industry applications. The proposed solution is technically sound and is well supported by the extensive evaluation presented. Overall, the paper is easy to follow, and the motivations are also well justified.

The experimental settings and the data preparation are well documented, the authors have performed extensive empirical studies against multiple public datasets and several baseline methods to show their proposed method’s efficacy. The studies consistently show the proposed Cold Brew solution to solve the SCS problem results in significant improvement – especially for the tail & isolation splits.

It should be noted that the focus is primarily on graph-based solutions for solving the SCS problem, which is not the only possible setup. Also, the evaluation and problem formulations are mainly focused on node label prediction (accuracy metric), it would have been interesting to see some metrics like hit-rate (especially for tail & isolated splits of e-commerce datasets) – for ranked recall generation(link-injection) using node representations learned from SCS for a cold-start item which is crucial for generating recommendation. 


summary_of_the_review: Overall , I think the paper studies an important and interesting problem and presents a good solution which is theoretically sound. It however could try to cover more broader and crucial task especially w.r.t cold-start recommendations .
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Cold Brew to distill the knowledge of a GNN teacher into an MLP student to handle the tail and cold start generalization problem by using the head part of the graph to guide the discovery of the latent neighborhoods of tail and isolation nodes. The paper also proposes a new metric to measure the contribution ratio of node features w.r.t. the adjacency structure. The experiments on several public datasets and a proprietary e-commerce graph show the effectiveness of the proposed method.
main_review: Strengths:
1. The paper is well-motivated. The problem studied in this paper is important for the graph domain.
2. The proposed knowledge distillation method sounds interesting and novel.
3. The paper proposes the feature-contribution ratio to guide the selection of model architectures.

Weaknesses:
1. The authors claim that the related works about cold start do not address the case of noisy or missing neighborhoods. However, the paper also focuses on the general cold-start problem and does not address the noisy or missing neighborhoods. I did not find any discussion or designed strategy to handle the noisy or missing neighborhoods explicitly.

2. It is not clear why the structural embedding can encode the label information.

3. The assumption behind the proposed knowledge distillation method is not clear. Can I suppose that the paper has an implicit assumption that is "the nodes with similar features should have similar neighborhoods"? Because the student is to learn a mapping from the node features to $\overline{E}$ that is learned graph structure.

4. The authors claim that the MLP student will behave like the GNN teacher but generalize better to tail and cold-start nodes. But as Table 3 shown, GCN+SE outperforms the student MLP in the tail scenario, and the results need more explanation and discussion.
summary_of_the_review: The paper provides an interesting and novel solution for the critical cold-start problem. However, several claims are not well-explained or well-supported.

The authors addressed most of my concerns, I would like to update my score.
review_title: null

Meta-review: The reviewers agree that the paper studies an important and interesting problem and presents a good solution which is theoretically sound. The paper can be further improved by looking into more applications such as cold-start recommendations.
Decision: Accept (Poster)