Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studied NAS algorithm for GNN. The author proposed to transfer the prior architectural design knowledge to the novel task of interest. The author first extracts scale-invariant measures based on Fisher Information Matrix (FIM). The scale-invariant measures are used to form the task feature, which are further projected to task embeddings. The task embeddings are learned via a ranking loss built on top of GraphGym's extact metric space. Experiments justified the importance of the task embeddings in the AutoTransferer algorithm. The AutoTransferer also outperforms baselines like GraphNAS, 

strength_and_weaknesses: Strength

1. The idea of utilizing FIM to produce task-feature and further finetune it with ranking loss is novel.
2. Ablation analysis justifies the effectiveness of the task embedding algorithm proposed by the paper.


Weaknesses

1. It seems that "[AAAI2021] One-shot graph neural architecture search with dynamic search space" also discussed task transfer. The author may need to compare with one-shot NAS or clarify the difference.
2. The author pointed out that there are three stages in GNN architectures: 1) pre-processing, 2) message-passing, 3) post-processing. The author only considered message-passing layers and pointed out that it is commonly regarded as the most important part of GNN. However, since different tasks adopt different pre-/post-processing modules, it is not convincing to directly state tat message-passing is the most important.
3. It will be good if the author can list the performance of SOTA algorithm in each dataset discussed in Table 1. This will help readers understand the gap between NAS models and human-engineered SOTA architectures.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy-to-understand.
Quality: Overall, the paper is technically sound. Weaknesses pointed out in the previous section.
Novelty: To the best of the reviewer's knowledge, the algorithm of obtaining task embedding is novel 
Reproducibility: The reviewer has not checked reproducibility but the author has provided enough details in the appendix.
summary_of_the_review: Voted for weak accept because the algorithm is novel. However, I still have concerns pointed out in the Weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel AutoML method, AutoTransfer, for graph learning tasks. It introduces a task-model bank that can transfer the model design to similar new tasks. The experiments indicate that the method is efficient and improves the performance of the found models.
strength_and_weaknesses: ### Strengths
1. The paper introduces an interesting method for transferring knowledge from a previously searched task to a new coming task with a task-model bank design.
2. It proposes an interesting task embedding with several random-initialized anchor models, and the experiments show that the task similarity is promising.
3. The methods are tested on multiple benchmarks and are proven to be efficient and performant.
4. It releases a large task-model dataset that can inspire future research.

### Weaknesses
1. How are the anchor models selected? I failed to find the architecture design of the anchor models, and it would be good to see how diverse the anchor models should be.
2. The graph architectures used in the paper are quite limited. For example, on the OGB leaderboard, many different GNN (non-GNN) algorithms are applied and achieve significantly better results. It would be interesting to have some of those models in the search space.
3. It would be interesting to see some examples of the models selected for a group of tasks, to get some insights of the model design.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The proposed methods are interesting.
summary_of_the_review: The paper proposes an interesting method for transferring knowledge in AutoML for graph learning tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper "AutoTransfer: AutoML with Knowledge Transfer - An Application to Graph Neural Networks" makes two major contributions: First, the authors devise a benchmark for searching graph neural networks, consisting of 12 task and comprising 120,000 evaluated task-model combinations. Second, they propose a method for transfer learning, suggesting a prior based on past experience for searching neural architectures on a new dataset.
strength_and_weaknesses: [+] The provided benchmark dataset is valuable and hopefully facilitates and enables research on transfer learning graph neural networks.

[+] The proposed transfer learning approach seems to perform reasonably well refining the state of the art.

[+] To the best of my knowledge, the proposed method is novel.



[-] The authors state that all GNNs can be described in terms of three stages: pre-processing with MLPs, message passing layers, and post-processing with MLPs. However, I would argue that this is quite restrictive and not able to reflect all possible architectures. See for example 

Lee, Junhyun, Inyeop Lee, and Jaewoo Kang. "Self-attention graph pooling." International conference on machine learning. PMLR, 2019.

and 

Ma, Yao, et al. "Graph convolutional networks with eigenpooling." Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019.

where pooling operations are added between the convolutions resulting in iterative changes of the graph structure. Of course, such a benchmark database needs to be restricted in some sense so that the set of candidates is reasonably limited. However, I would suggest to rephrase the statement in Section 4.2 a little bit so that it makes clear that the benchmark limits itself to such GNNs.

[-] It is not clear why the authors chose the mentioned datasets for evaluation and why results on the other datasets are not reported.
clarity,_quality,_novelty_and_reproducibility: - There is already another transfer learning approach named "Auto-Transfer" and frankly speaking the name is not very innovative or self-describing: Murugesan, Keerthiram, et al. "Auto-Transfer: Learning to Route Transferrable Representations." arXiv preprint arXiv:2202.01011 (2022).
- Why is only Kendall's correlation coefficient considered? There could be different degrees of similarity, something like weak similarity where only a top-k ranking needs to be consistent. However, since one is not interested in the precise ranking of low performers, one could argue that this is less relevant for comparing similar datasets. Maybe a correlation measure such as normalized discounted cumulative gain (NDCG) would be a better choice here? The authors state that they see a justification for Kendall's tau in the results but do not discuss other ranking correlation measures.

### Clarity
The clarity is overall quite good. Language is clear and the paper is well structured.

### Quality and Novelty
The quality of the paper is very good and the approach is novel - at least to the best of my knowledge.

### Reproducibility
Important details regarding the methods evaluated are missing. In particular how competitors have been parameterized, what kind of evolution was used as a baseline and how it was configured etc.
summary_of_the_review: The paper makes good contributions to the field and provides some new artefacts for the community which can be quite useful to foster future research on NAS for GNNs. The paper has some minor issues which can however be fixed quite easily. Therefore, I would recommend to accept the paper under the condition that the issues are fixed for the camera ready version.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a neural architecture search framework for GNNs, which retrieves the most similar GNN architectures from a large database of randomly initialized GNNs based on their task-level similariies. The task-level similarities of the GNNs are computed with a modified version of Fisher Information Matrix embedding [Archille et al. 19] to deal with the irregularities in the graph data. The experimental results verify the effectiveness of the proposed framework.

All reviewers leaned toward acceptance, as they considered the idea of retrieving neural architectures based on the task similarity encoded using Fisher Information Matrix as novel, the construction of the large database of GNNs for transferable NAS as useful for future research, and the proposed framework to be practically effective. However, the reviewers also had concerns regarding not considering other aspects that can have large impact on the GNN models (e.g. pooling), comparison against state-of-the-art NAS approaches, and lack of discussion of closely related works.

Despite the overall positive sentiment among the reviewers, I am concerned about the lack of novelty in the general AutoML perspective, and the overclaimed contributions of the work, which I believe is due to insufficient literature study.

- The idea of retrieving an architecture from a model zoo based on the task similarity is not novel, since [Jeong et al. 21] proposed the same idea, with a different task encoding. The authors argue that [Jeong et al. 21] cannot distinguish between two tasks, but the embedding simply has less discriminative power and the high-level ideas are essentially the same.

- The idea of using Fisher Information Matrix to encode task similarity is the contribution of [Archille et al. 19], and using Fisher Information Matrix for NAS has been already done in [Le et al. 22].

- However, the authors did not discuss these works, which are either conceptually similar or essentially based on the same method for a non-graph domain. The title “AutoTransfer: AutoML with Knowledge Transfer - An Application to Graph Neural Networks” seems to suggest the method is applicable to non-graph domains, but there is no concrete formulation or experimental results on other domains. 

I believe that the contribution of the proposed work is strictly limited to the extension of them to the graph domain. 
Construction of a large-scale GNN database
Modification of the FIM-based task embedding for encoding graphs.

Yet, the title of the paper, the name of the proposed framework, and the claimed contributions of the authors do not accurately convey what are actually done. In the final revision of the paper, the authors should clearly acknowledge that similar ideas have been proposed in non-graph domains and revise the title, naming, and the claimed contributions accordingly. 

[Archille et al. 19] Task2Vec: Task Embedding for Meta-Learning, ICCV 2019.
[Jeong et al. 21] Task-Adaptive Neural Network Search with Meta-Contrastive Learning, NeurIPS 2021
[Le et al. 22] Fisher Task Distance and Its Application in Neural Architecture Search, IEEE 2022

justification_for_why_not_higher_score: The contribution of this paper is strictly limited to the application of existing ideas to the graph domain due to the following reasons:
- The idea of retrieving architectures from a large model zoo is not novel, since it has been already done in [Jeong et al. 21]. 
- The idea of using Fisher information matrix for neural architecture search was proposed in [Le et al. 22].
justification_for_why_not_lower_score: The proposed framework as well as the large-scale GNN database can serve as good benchmark and baselines for future research on NAS for GNNs.
Decision: Accept: poster