Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose PARETOGNN, which is a multi-task SSL framework using the concept of Pareto optimality.


The self-supervised learning (SSL) for multi-task is still challenging especially over graph-structured data.


In this environment, the SSL frameworks for the graph that adhere to just one objective might perform poorly to the others.


To improve task generalization, PARETOGNN is simultaneously trained on various SSL tasks while promoting Pareto optimality using the Multiple Gradient Descent Algorithm(MGDA).


It allows the model to capture inherent patterns that are relevant to several pretext tasks while reducing possible conflicts.


As a result, the model learns different patterns transferable to multiple tasks on graphs.
strength_and_weaknesses: **Strengths**


- The paper including the appendix is well-written and organized. 

- The authors compare several features of the existing methods with the proposed method and provide a detailed discussion.

- The results on benchmarks are comparable to or better than most baselines.

---


There has been papers that explore Pareto optimality for SSL or MTL [1,2].

What is the main contribution of your work compared to them?

Of course I could have noticed, but I'm asking for the benefit of other readers.

Because it might seem to just simple extension of these series of works into graph version.

In Pareto MTL [2], the authors claim that the multiple gradient descent algorithm (MGDA) is not capable to incorporate different trade-off preference information as illustrated in Figure 2 of [2].

Does it mean ParetoGNN also can have the same weakness?

---


[1] Pareto Self-Supervised Training for Few-Shot Learning, Chen et al., CVPR 2021

[2] Pareto Multi-Task Learning, Lin et al., NeurIPS 2019
clarity,_quality,_novelty_and_reproducibility: I asked novelty issue on the above section.
summary_of_the_review: I set the confidence of this review to a low score because I would change (or finalize) my decision after the discussion period (regarding the novelty issue).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-task self-supervised learning framework namely ParetoGNN for node representation learning on graphs. The proposed method adopts five pretext self-supervised learning tasks and designs a multiple-gradient descent algorithm to reconcile different tasks by actively learning and minimizing conflicts. Extensive experiments over four downstream tasks and 11 benchmark datasets show that the proposed method can improve the model's task generalization ability.
strength_and_weaknesses: Pros:  
[+] Multi-task SSL for node representation learning over graphs is a novel and important problem.   
[+] This paper shows that task conflicts commonly exist in real-world graph datasets and theoretically proves that the proposed method can resolve task conflicts.  
[+] This paper conducts extensive experiments over four downstream tasks and 11 benchmark datasets.  
[+] The illustration of motivation, method design, and experiments are quite clear.  
[+] Experiment details are contained in the appendix, including hyperparameters, settings, etc. Source codes are also included.  

Cons:  
[-] It seems that the proposed method to resolve task conflicts is agnostic of domains, i.e., not specifically designed for graphs. It would make the paper stronger if further analysis of the difference between conflicts in graphs and other domains could be analyzed. For this reason, I think that the technical novelty is somewhat limited.  

=== after rebuttal ===
I have read the rebuttal and thank the authors for further clarifications. All things considered (including the quality of my other reviewed papers), I'd like to keep a slight positive score as 6.

clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility are good, while the novelty is somewhat limited. 
summary_of_the_review: See above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the use of multi-task learning for GNN self-supervised learning and particularly uses a combined gradient based on pareto optimality. The model seems to improve upon existing ones, interestingly, especially also in terms of generalization capability w.r.t. heterophily/homophily.

strength_and_weaknesses: (+) The question represents an interesting topic to be investigated and the paper presents a reasonable approach which, to the best of my knowledge, is novel, especially the integration of the gradients.

(+) The authors provide theory proving convergence. Please note that I did not check all that in full detail.

(+) The evaluation considers various datasets, also challenging ones from OGB, tasks, and baselines. I am no expert in SSL to be able to judge if the most important competitors are considered.

(-) Some statements seem unlucky/misleading to me:
- The main question "Does combining multiple philosophies enhance task generalization for SSL-based GNNs?" sounds a bit too easy. "Yes" should be a very likely I would think. 
- Similarly, the authors claim throughout the paper that models focusing on a single task perform "poorly" (as narrow experts) and generalization beyond a single task is needed. I would think that these are too bold claims. If we developed perfect experts over time, this would not be a problem. Nevertheless, the fact that the paper shows that the generalization obtained helps improving over single-task-focused models is great. I just would frame it differently.

(-) The paper is missing related work. For example, [1] propose to combine different self-supervised learning strategies and demonstrate that the combination is beneficial. Similarly, I would consider multi-task learning in GNNs as related work to be mentioned in more detail.
p.9 "In these frameworks" - if there are more, these should be referenced as well since this is closely related work.

[1] Hu et al. STRATEGIES FOR PRE-TRAINING GRAPH NEURAL NETWORKS, ICLR 2020


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and to me seems to tick off the quality and novelty requirements. Code is also provided.
summary_of_the_review: Altogether, this work seems to be a solid contribution to me. I am too few expert of self-supervised learning to judge if it is really novel w.r.t. SOTA. If the other reviewers can confirm the latter, I would suggest acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I. Summary:

- I.1 Investigated Problem:
The paper investigated the problem of task generalization for self-supervised learning (SSL) based graph neural networks (GNNs) where it shows that the promising performance on 1 or 2 tasks of SSL-based GNNs does not translate into good task generalization across various downstream tasks and datasets. ParetoGNN is then introduced for node representation over graphs.

- I.2 Proposed Solution:
The proposed method (ParetoGNN) is self-supervised by manifold pretext tasks that observe multiple philosophies. These multiple philosophies are reconciled using a multiple-gradient descent algorithm promoting Pareto optimality. The algorithm captures patterns that are relevant to several pretext tasks which reduce potential conflicts and in this way, knowledge is transferable to multiple tasks on graphs.

- I.3 Validity Proof of the Proposed Solution:
Both theoretical and empirical evidence is provided to support the validity of the proposed solution as a convergence guarantee of the presented algorithm is provided as well as extensive experiments over four downstream tasks and 11 benchmark datasets. The showcased evidence demonstrates that the presented solution can improve the model's task generalization ability.

II. Strengths:

- II.1 From a structural point of view: Reviewers agreed upon the organization and writing quality. 
- II.2 From an analytical point of view: Reviewers pointed out the clarity of the submitted paper from a general perspective. More precisely. Reviewers appreciated:
    . The clarity of the motivation;
    . The presentation of the designed method;
    . The theoretical and empirical evidence provided to support the case of the proposed solution;
    . The discussion related to the comparison conducted with several features of the existing methods with the proposed method. In fact, it has been shown that the presented solution is comparable to or better than most baselines;
    . The transparency aspect of the submission as open source code is provided for reproducibility purposes and hyperparameters settings are detailed in the appendix.
- II.3 From a perspective of soundness (development, unity, and coherence) and completeness (correctness): The strength points mentioned above are sufficient evidence of the soundness and completeness of the paper. An additional point reinforcing the strength points mentioned above is the active interaction of the authors during the rebuttal period and their openness to concerns and questions raised by the reviewers. The openness followed by the active interactions and persistence in answering questions and addressing concerns related to the submission demonstrate the author's intellectual honesty and good faith in conveying the details of the proposed solution to the reader as well as its benefits. Such a rebuttal is a great example of how can rebuttals be conducted to reinforce a presented idea in a paper.

III. Addressing what can be thought of as weaknesses:

The weaknesses pointed out by the reviewers relate mostly to the potential aspect of the proposed method and its novelty.
- III.1 Before the rebuttal period, reviewers raised concerns about the novelty of the proposed method which is related to the fact that ParetoGNN cannot incorporate pre-defined trade-offs between pretext tasks. The authors addressed these concerns by explaining in which context it is useful to use preference vectors. The main focus of the paper is task generalization and the authors provided arguments and empirical results to prove that the use of preference vectors would hurt generalization across datasets and tasks.
- III.2 Missing related work suggested by the reviewers has been added by the authors.
- III.3 Another point has been raised by a reviewer which is about the generic aspect of the proposed method in question and its analysis beyond GNN models. This was discussed with one of the reviewers during the AC-reviewers meeting and we agreed that any good / idea can be applied to more a general context. This can be considered as a strong benefit of the proposed approach as it can be investigated in other contexts of applications and the use of models.

Most of the points that could be thought of as weaknesses have been addressed and it has been clarified that what can be thought of as a weakness is not. Unanimously, the reviewers agree on the acceptance of the submission.

IV. Potential of the paper:

- IV.1 From a Potential perspective (Potential of the paper to the community): The proposed solution has a great potential to be of benefit to the whole community, especially the ones interested in multi-objective learning and graph neural networks. The extension and investigation to other models can also be of great benefit to the representation learning community in general.
justification_for_why_not_higher_score: All reviewers agree that the paper in question is above the threshold of acceptance. Unanimously, the paper reviewers graded the score of 6 which a slightly above the threshold of acceptance. This being said, the paper has the potential to be of great benefit to the community and its value can be appreciated by researchers investigating the problems addressed in the submission.
justification_for_why_not_lower_score: All reviewers recommend the acceptance of the paper. The proposed method has the potential to be of great benefit to the community.
Decision: Accept: poster