Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a new auxiliary learning method to improve the existing GCNs. Considering that the edges in many graphs are unweighted and nodes labels are often one-hot, the authors added two auxiliary tasks to enrich the topology information in the graph node classification task. One is link prediction which generates probabilistic edges; and the other is to generate soft labels. The probabilistic edges and soft labels are iteratively updated by a meta auxiliary learning strategy in a smart way, and the results on three standard benchmark datasets outperform some baselines.

main_review: The motivation of the paper is reasonable but maybe not so significant. It is limited to the node classification task on graphs with binary edges. In practice many graphs have weights or attributes on edges, and the application of most GNNs are also not limited to the node classification task. 

The paper combines the ideas of auxiliary learning and meta pseudo labels to enrich the topology information. It is novel in the context of graph learning, but some concepts or descriptions are confusing. For example, I am not sure whether the label generation can be regarded as an auxiliary task, since g_\phi is a totally different model which does not share any parameters with the target model in the main task.

The experiments are weak. As the area of deep graph learning is developing so fast, just using the three small graph datasets with standard settings are not enough nowadays. It cannot give us enough insights why and how the proposed method works. For example, 1. the advantage of using probability edges instead of original ones may be making the model not only more expressive but also robust. Can you make some perturbations on the edges and see how the performance changes? 2. The utilization of teacher model and pseudo labels maybe make the unlabeled data more efficiently used, so can you change the proportion of labeled/unlabeld data  to show the power of your model? Or can you experiment on larger datasets such as OGB? 
This kind of experiments will make the paper more solid. Otherwise, it is not convincing. 

limitations_and_societal_impact: 
I do not see any negative social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel meta-learning framework that utilizes auxiliary learning tasks of link predictions and label generations to improve the performance of GCNs on node classifications tasks. The meta-learning algorithm adopts the idea from [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://arxiv.org/abs/1703.03400) so that the models for auxiliary tasks are updated in a way that has higher impacts on the primary task.  The authors use two self-supervised learning tasks, edge predictions and label generation, as the auxiliary tasks.  The method performs favorably in the three standard public graph data sets with higher prediction accuracy.
main_review: This paper is generally well written and well structured. I appreciate the authors' efforts to keep the discussions organized and easy to understand. The experiment's setting is reasonable and the results of the experiment suggest that the proposed method is able to achieve higher prediction accuracy. And the design choices are properly examined in the ablation study.

Using self-supervised tasks as auxiliary learning tasks in GNNs has been discussed in a few pieces of literature recently, such as [1][2]. Different from prior arts, this paper presents a novel meta-learning algorithm to the GNNs that using the impacts on the primary tasks to direct the update of the auxiliary tasks. Thus the primary tasks might suffer less from degeneration, which is common for multi-task learning.  However, I am curious about the difference between this method and the one in [3], which is general auxiliary learning using self-supervised tasks and a similar meta-learning. In general, I find the contribution of this paper quite incremental.


[1] Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes

[2] When Does Self-Supervision Help Graph Convolutional Networks? 

[3] Self-Supervised Generalisation with Meta Auxiliary Learning, NeurIPS 2019.

limitations_and_societal_impact: The limitations and potential negative sociteal impacts are not addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, they introduced a method for node classification in graphs using Graph Convolutional Networks (GCNs) in multi-task fashion to address incomplete annotation problem. Two auxiliary tasks, link prediction and label generation, are trained with the target task (node classification) simultaneously using a graph meta-learning approach. 

main_review: 


Strengths:

- In general, paper is well-written and well-organized; e.g.,  Algorithm 1 is self-descriptive. There are some small confusion for me that I mention later.

-  I like the idea of using two related auxiliary tasks to improve node classification problem; specifically, they used two link prediction and label generation tasks, in which they leveraged graph-structured information to improve the performance of target task (node classification).

- The experiments are thorough, they cover both qualitative and qualitative analysis, the ablation studies were interesting

Notes:

- Figure 1 is a little bit confusing for me. I would suggest using the citation network example instead of image example in the figure 1 to be consistent with the experiment data.

- For figure 2 and 3, I would recommend adding explanation related to color-coded features and nodes.for example, what green and blue colored nodes mean? it will be also better if they could add some explanation in the captions of tables.

- In the link prediction task, if I understand it correctly, they used the similarity of every two node to compute if edge is between them? What is the threshold to infer there is an edge between two nodes?

- In the link prediction task, it would be useful to mention the intuition why they use a sample of edges for each iteration? Is it related to time or complexity issues?

- In equation 8, for calculating the loss, I think for the psudo labels, you could use the cross entropy loss instead of MSE. What is the intuition of using MSE?

- In section 4.3.2, I do not understand the difference between "w/o-oriG" and "AL-GCN". My assumption is that both of them use the reconstructed adjacency matrix for backbone model. What is the difference between these two?  

- In comparison with the other node classification methods, I would recommend adding a new paraphrase explaining why we should use "AL-GCN" instead of other methods; basically what is the winner characteristics of this method comparing others. Does this have better time complexity? Or does it perform better in semi-supervised setting? I understand it outperforms previous ones with respect to accuracy, but we could argue other benefits of using this approach

- I think this framework could be model-agnostic. I means we will be able to use GATs instead of of GCN? This will be nice future work.

- In line 56, one extra "sudo"?

**update: After receiving authors' response, I would like to appreciate their time for addressing my concerns. My evaluation on the paper is reflected after reading their response.

limitations_and_societal_impact: As far as I understand, for the limitations, they mentioned that they plan to address the noise in the data by designing more robust method.

They did not mentioned any potential negative societal impact in the work; and I do not see there could be any negative societal impact in this work as well.
review_title: null

Meta-review: The paper has merit and received mixed reviews, but overall it is too incremental. I think there are several avenues along which the paper can be strengthened, some of which are mentioned in the discussion. Pursuing those, it seems likely to get to a paper with critical mass for a top tier ML conference. The submitted paper is not quite there yet.
Decision: Reject