Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The proposed work looks at the task of automated self-supervised learning (SSL) on graphs, by using pseudo-homophily as a surrogate objective combined with a search strategy for the proposed approach, AutoSSL . 

Homophily is defined as the average of sameness of labels over pairs of connected vertices. Pseudo-homophily is computed by assigning labels based on k-means clustering. The work theoretically shows that maximizing pseudo-homophily is shown to maximize the upper bound of mutual information between pseudo-labels and downstream labels. 

As cluster assignments are not differentiable and need search over a large space, the work looks at using evolutionary strategies (AutoSSL-ES) and differentiable search through soft cluster assignments using Gaussian mixture model (AutoSSL-DS).

Experiments are performed on 5 SSL tasks and 8 datasets. AutoSSL based approaches outperform several other unsupervised baselines and perform comparably to supervised baselines when measured using normalized mutual information, accuracy of node classification and psuedo-homophily. 
main_review: Strengths:
1. Paper is well written and easy to follow.
2. Pseudo-homophily is shown to be a good surrogate measure for mutual information, and combining with search strategies proves effective on several benchmark datasets and tasks.


Weaknesses:
1. Method is somewhat simple and intuitive
2. Method underperforms significantly on clustering task for Photos dataset. 
summary_of_the_review: The method is well motivated and quite intuitive, the paper is easy to follow. Extensive experimental results are provided on several tasks and datasets, but statistical significance of improvements is missing, which is needed given large confidence intervals reported.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper mainly studies the problem of automatically weighting multiple self-supervised tasks on a graph without information of true labels, in order to make downstream node-level prediction tasks perform better. To overcome the challenge of missing ground truth in pre-training tasks, the authors propose to use a property named "homophily" of the graph to create pseudo labels. Then, they propose two algorithms to automatically adjust weights among different tasks: one is based on evolution algorithms; another is based on gradient descent. Extensive experiments on different real-world datasets show the improvement of the proposed methods compared with a single pretext task.
main_review: Strengths:
The paper is well-written and easy to follow. I can easily understand the idea of this paper.

This paper conducts extensive experiments on 7 different datasets from different domains to show their improved performances, which is very reasonable and convincing to me.

Weaknesses:
The main concern I have towards this paper is its novelty. I can see there is much effort of this paper to make adjusting different pretext tasks on a graph, especially the authors bring up the important "homophily" property of graph to create pseudo labels for self-supervision. However, the essence of this paper is still dynamically adjusting weights for different losses, as the authors mentioned in the section Related Work of Automated Loss Function Search. There is so much work on reweighting different loss functions which have been well studied. Though the authors mentioned that "the problem of self-supervised loss search for graphs remains rarely explored", I find out the solved problem in this paper is actually an old problem, which is limited in novelty and significance.

The core contributions of this paper mainly include (i) making use of homophily to create pseudo labels for self-supervision; (ii) proposing AutoSSL-ES based evolutionary strategy (iii) proposing AutoSSL-DS based on meta-gradient descent. For (i), after checking the paper and appendix, I think it is good and reasonable. But why should the authors propose two different methods to solve the problem? The author mentions AutoSSL-ES requires evaluating a large population of candidate combinations which is not practical and AutoSSL-DS is much efficient. So, what are the advantages of AutoSSL-ES? The authors should give more explanations and analysis towards choosing these two methods. Otherwise, I would consider they are simply a combination of existing works without much contribution.

I observe that the improvement of AutoSSL compared with the best result of individual tasks is not that significant. Moreover, the best results of individual tasks are from PAR and DGI. This leads to a question that does is really needs to weigh these 5 tasks. Maybe considering the other 3 tasks is not that helpful.

Also, more recent works of graph self-supervised learning can be considered and cited. For example, Han X, Huang Z, An B, et al. Adaptive Transfer Learning on Graph Neural Networks[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 565-574.
summary_of_the_review: Though this paper is well-written and studies an important and popular problem in graph learning. I still have concerns in different aspects.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Self-supervised learning (SSL) methods for graphs take a given graph with node attribute information and construct various SSL tasks using structural and attribution information. These tasks provide self-supervision for training graph neural networks without accessing any labeled data. Existing studies show that different SSL tasks can lead to varying downstream performance across tasks, suggesting that the success of SSL tasks strongly depends on the dataset and the downstream task. This observation is important, yet not surprising, as similar findings were made in other data modalities (text, vision). As a result, selecting the 'right' SSL tasks can be important, which serves as a motivation for this study. 

This study develops an approach (called AutoSSL) for combining multiple SSL tasks for unlabeled representation learning. To this end, it defines a pseudo-homophily metric to measure the quality of learned representations. Based on the pseudo-homophily, the study describes two techniques to search & combine SSL tasks, one using an evolutionary algorithm and the other using meta-gradient descent. The AutoSSL approach is evaluated on eight datasets considering combinations of five SSL tasks (one contrastive learning task and four predictive tasks). 
main_review: 
(1) Performance gains are small or non-significant. Closely examining Tables 1-3, I see many scenarios where the best performing AutoSSL model performs on par with the strongest baselines or offers minuscule improvements. (that is, when taking into account standard variation of performance across independent runs). For example, in Table 2, results for CoraFull are 61.10±0.68 vs. 60.56±0.33 (= on-par performance), results for Physics are 95.57±0.02 vs. 94.66±0.10 (= minuscule improvement, which we only see after considering the stronger AutoSSL-ES variant, AutoSSL-DS performs at 95.13±0.36, which leads to essentially no performance). There are many such scenarios in Tables 1-3. 

(2) The AutoSSL approach is concerned with finding an effective way to combine individual SSL tasks (problem definition in Eq. 1, where $\mathcal{H}$ is set to the negative pseudo-homophily). That is, it assumes that a set of $n$ SSL tasks are already given and it will find a good combination of those tasks formulated as a weighted addition of individual SSL tasks, each task associated with a learned task weight $\lambda_i$. Such formulation seems quite restrictive. It does not allow for any interaction between the loss functions. Further, it also assumes that the user already knows what are potentially good SSL tasks. 

(3) It is unclear how the proposed strategy could be used for other contexts, such as link prediction and graph classification.
summary_of_the_review: Both AutoSSL variants (evolutionary strategy and the gradient descent version) seem ad-hoc and straightforward combinations of existing tools. This is not a problem on its own, however, the empirical gains are not convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors use homophily datasets and meta learning. For self-supervised learning, authors use evolutionary strategy.
main_review: 
1. "Is Homophily a Necessity for Graph Neural Networks?" is a paper. In that paper, it states Homophily is not necessity. If dataset is Homophily, the task comes to easy. In this paper, authors use such easy datasets.
2. If the meta-learning is the contribution, it is not a contribution of the paper but is a tool.
3. The experimental results show it obtains better performance.
4. Is the "evolutionary strategy" a contribution? 

summary_of_the_review:  homophily datasets is used and authors shows theoretical analysis.
review_title: null

Meta-review: Strengths:
* Strong empirical study across multiple datasets. However, the gains are not as impressive as for other pretraining domains, such as text or images.
* Interesting formulation of pseudo-homophily as an objective to optimize in the self-supervision stage
* Well-written paper

Weaknesses:
* Novelty may be limited by the fact that the method is essentially learning (or searching) for a weighted average of self-supervised training objectives
* In that case, while the pseudo-homophily angle is interesting, there may be other appropriate baselines for yielding this weighted combination of tasks that are not explored
* There is concern about the degree of empirical improvements on certain datasets
Decision: Accept (Poster)