AutoTransfer: AutoML with Knowledge Transfer - An Application to Graph Neural Networks
Authors: Kaidi Cao
Jiaxuan You
Jiaju Liu
Jure Leskovec
Venue: ICLR 2023 poster (2023)
Keywords: Graph Neural Networks
AutoML
Knowledge Transfer

Abstract: AutoML has demonstrated remarkable success in finding an effective neural architecture for a given machine learning task defined by a specific dataset and an evaluation metric. However, most present AutoML techniques consider each task independently from scratch, which requires exploring many architectures, leading to high computational cost. Here we propose AutoTransfer, an AutoML solution that improves search efficiency by transferring the prior architectural design knowledge to the novel task of interest. Our key innovation includes a task-model bank that captures the model performance over a diverse set of GNN architectures and tasks, and a computationally efficient task embedding that can accurately measure the similarity among different tasks. Based on the task-model bank and the task embeddings, we estimate the design priors of desirable models of the novel task, by aggregating a similarity-weighted sum of the top-K design distributions on tasks that are similar to the task of interest. The computed design priors can be used with any AutoML search algorithm. We evaluate AutoTransfer on six datasets in the graph machine learning domain. Experiments demonstrate that (i) our proposed task embedding can be computed efficiently, and that tasks with similar embeddings have similar best-performing architectures; (ii) AutoTransfer significantly improves search efficiency with the transferred design priors, reducing the number of explored architectures by an order of magnitude. Finally, we release GNN-Bank-101, a large-scale dataset of detailed GNN training information of 120,000 task-model combinations to facilitate and inspire future research.

Published as a conference paper at ICLR 2023
AUTOTRANSFER: AUTOML WITH KNOWLEDGE TRANS- FER AN APPLICATION TO GRAPH NEURAL NET-
WORKS
Kaidi Cao
Jiaxuan You
Jiaju Liu
Jure Leskovee
Department of Computer Science, Stanford University
(kaidicao, jiaxuan, jiajuliv, jure) êcs. stanford.
ABSTRACT
AutoML has demonstrated remarkable success in finding an effective neural ar- chitecture for given machine learning task defined by specific dataset and an evaluation metric. However, most present AutoML techniques consider each task independently from scratch, which requires exploring many architectures, leading to high computational costs. Here we propose AUTOTRANSFER, an AutoML solu- tion that improves search efficiency by transferring the prior architectural design knowledge to the novel task of interest. Our key innovation includes a task-model bank that captures the model performance over a diverse set of GNN architectures and tasks, and computationally efficient task embedding that can accurately mea- sure the similarity among different tasks. Based on the task-model bank and the task embeddings, task, by aggregating a monilarity-weiihhee tasks that are similar the task of interest. The computed design priors can be used with any AutoML search algorithm. We evaluate AUTOTR ANSFER on six datasets in the graph machine learning domain. Experiments demonstrate that our proposed task embedding can be computed efficiently, and that tasks with simi- lar embeddings have similar best- rest-performing architectures; (ii) AUTOTRANSFER significantly improves search efficiency with the transferred design priors, reducing the number explored architectures by an order of magnitude. Finally, we release GNN-BANK 1-01- 1, a large-scale dataset of detailed GNN training information of 120,000 task-model combinations to facilitate and inspire future research.
estimate the design priors of desirable models of the novel sum of the top-K design distributions
INTRODUCTION
Deep neural networks are highly modular, requiring many design decisions to made regarding network architecture and hyperparameters. These design decisions form search space that is nonconvex and costly even for experts to optimize over, especially when the optimization must be repeated from scratch for each new case. Automated machine learning (AutoML.) is active research area that aims to reduce the human effort required for architecture design that usually covers hyperparameter optimization and neural architecture search. AutoML has demonstrated success (Zoph and Le 2016) Pham et al. 2018 Zoph et al.) 2018: Cai et al. 2018 He et al. 2018
Guo et al. 2020 Erickson et al.) 2020 LeDell and Poirier (020) many application domains.
Finding a reasonably good model for a new learning tas in computationally efficient manner is crucial for making deep learning accessible to domain experts with diverse backgrounds. Efficient AutoML is especially important in domains where the best sensitive to the task. notable example the domain of graph learning First, graph learning methods receive input data composed of variety of data types and optimize over tasks that span an equally diverse set of domains and modalities such as recommendation (Ying et al. 2018 He let al. (2020). physical simulation Srrechez-Gnzzlezzz et al. 2020; Plaff et al. 2020), and bioinformat- ics (Zitnik et al. 2018). This differs from computer vision and natural language processing where the
are highly
In this paper, we refer to a task as given dataset with an evaluation metric/loss, e.g., cross-cntropy loss on node classification on the Cora dataset.
We focus the graph learning domain in this paper. AUTOTRANSFER can he generalized other domains.

---

Published as a conference paper at ICLR 2023
input data has predefined, fixed structure that can be shared across different neural architectures. Second, neural networks that operate on graphs come with a rich set of design choices and a large set of parameters to explore. However, unlike other domains where few pre-trained architectures such as ResNet (He et al. 2016) and GPT-3 (Brown et al. 2020) dominate the benchmarks, it has been shown that the best graph neural network (GNN) design is highly task-dependent (You et al. 2020).
Although AutoML. as a research domain is evolving fast, existing AutoML solutions have massive computational overhead when finding a good model for new learning task is the goal. Most present AutoML techniques consider each task independently and in isolation, therefore they require redoing the search from scratch for each new task. This approach ignores the potentially valuable architectural design knowledge obtained from the previous tasks, and inevitably leads to high computational cost. The issue especially significant in the graph learning domain Gao et al. (201 /Zhou et al. (2019). due to the challenges of diverse task types and the huge design space that are discussed above.
Here we propose AUTOTRANSFER an AutoML solution that drastically improves AutoML archi- tecture search by transferring previous architectural design knowledge to the task of interest. Our key innovation is to introduce a task-model bank that stores the performance of a diverse set of GNN architectures and tasks to guide the search algorithm. To enable knowledge transfer, we define a task embedding space such that tasks close in the embedding space have similar corresponding top- uop-prformming architectures. The challenge here is that the task embedding needs to capture the performance rankings of different architectures on different datasets, while being efficient to compute. Our innovation here is embed task by using the condition number of its Fisher Information Matrix of various randomly initialized models and also a learning scheme with an empirical generalization guarantee. This way we implicitly capture the properties of the learning task, while being orders of magnitudes faster (within seconds). We then estimate the design prior of desirable models for the new task, by aggregating design distributions on tasks that are close to the task of interest. Finally, we initiate hyperparameter search algorithm with the lask-infommed design prior computed.
We evaluate AUTOTRANSFER on six datasets, including both node classification and graph clas- sification tasks. We show that our proposed task embeddings can be computed efficiently and the distance measured between tasks correlates highly (0.43 Kendall correlation) with model performance rankings. Furthermore, we present AUTOTRANSFER significantly improves search efficiency when using the transferred design prior. AUTOTRANSFER reduces the number of explored architectures needed to reach target accuracy by an order of magnitude compared to SOTA. Finally, we release GNN MONN-ANK--BA 101 101-1he first large-scal scale database containing detailed performance records for 120,000 task-model combinations which were trained with 16, 128 GPU hours-to to facilitate future research.
RELATED WORK
In this section, we summarize the related work on AutoML regarding its applications on GNNs, the common search algorithms,, and pioneering work regarding transfer learning and task embeddings.
AutoML for GNNs, Neural architecture search (NAS), a unique and popular form of AutoML for deep learning, can be divided into two categories: multi- -rial NAS and one-shot NAS. During multi-trial NAS, each sampled architecture is trained separately. GraphNAS (Gao et al. 2020) and Auto-GNN (Zhou et al.) (219)) are typical multi-trial NAS algorithms on GNNs which adopt an RNN controller that learns to suggest better sets of configurations through reinforcement learning.
Li et al. 202Ij) involves encapsulating the entire model space in one super- model, training the super-model once, and then iteratively sampling sub-models from the super-model to find the best one. In addition, there is work that explicitly studies fine-grained design choices such as data augmentation YYou al.] 2021), message passing layer type (Cai et al., 2021 Ding et al.) 2021 Zhao et al. 2021). and graph pooling (Wei et al. 2021). Notably,
One-shot NAS (e.g.. (Liu et al. 2018 Qin et al. 202 I
is the first AutoML solution for GNNs that efficiently transfer design knowledge across tasks.
HPO Algorithms. Hyperparameter Optimization (HPO) algorithms search for the optimal model hyperparameters by iteratively suggesting a set of hyperparameters and evaluating their performance. Random search samples hyperparameters from the search space with equal probability. Despite not
Source code is available htt.:p/////gciicooooooooooaaoaar
2

---

Published as a conference paper at ICLR 2023
Task Embedding Space
Design Distributions
Task-Model Bank
Agg
(historical performance records)
Task
Agg
Dim
Epoch
Val. loss
Dim
sum
Epoch
Cora
mean
128
200
0.26
Novel Task
TU-DD
sum
64
200
Interest
TU-DD
mean
128
200
0.86
Agg
TU-DD
max
258
800
0.52
Dim
Arxiv
200
0.68
Epoch
Figure 1: Overview of AUTOTRANSFER, Left: We introduce GNN-BANK- B-A---ANN--LLI large database containing diverse set of GNN architectures and hyperparameters applied to different tasks, along with their training/evaluation statistics. Middle: We introduce a task embedding space, where each point corresponds to a different task. Tasks close in the embedding space have similar corresponding top-performing models. Right: Given a new task of interest, we guide the AutoML search by referencing the design distributions of the most similar tasks in the task embedding space.
learning from previous trials, random search is commonly used for its simplicity and is much more efficient than grid search (Bergstra and Bengio, 2012). The TPE algorithm (Bergstra et al.) 2011) builds a probabilistic model of task performance over the hyperparameter space and uses the results of past trials to choose the most promising next configuration to train, which the TPE algorithm defines as maximizing the Expected Improvement value (Jones) 200T). Evolutionary algorithms (Real et al.) 2017: Jaderberg et al.) 2017) train multiple models in parallel and replace poorly performing models with "mutated" copies the current best models. AUTOTRANSFER is a general AutoML solution and can be applied in combination with any of these HPO algorithms.
Transfer Learning in AutoML. Wong et al. (018)) proposed to transfer knowledge across tasks by reloading the controller reinforcement learning search algorithms. However, this method assumes that the search space different tasks starts with the same learned prior. Unlike AUTOTRANSFER, it cannot address the core challenge in GNN AutoML: the best GNN design is highly task-specific. GraphGym (You et al.) 2020) attempts to transfer the best architecture design directly with metric space that measures task similarity. GraphGym (You et al. 2020) computes task similarity by training a set of 12 "anchor models" convergence which is computationally expensive. In contrast, AUTOTRANSFEE designs weihh task embeddings requiring minimal computations overhead. Additionally, Zhao and Bilen (202I): Li et al.| (202I) proposes to conduct architecture search on proxy subset of the whole dataset and later transfer the best searched architecture on the full dataset. Jeong et al. (202T) studies a similar setting vision domain.
Task Embedding. There is prior research trying to quantify task embeddings and similarities. Similar to GraphGym, Taskonomy (Zamir et al. 2018) estimates the task affinity matrix by summarizing final losses/evaluation metrics using an Analytic Hierarchy Process (Saaty) 1987). From a different perspective, Vesk2Vec (Achille al. 2019) generates task embeddings for given task using the Fisher Information Matrix associated with a pre-trained probe network. This probe network is shared across tasks and allows Ves estimate the Fisher Information Matrix of different image datasets. Le et al. (2022) extends a similar idea to neural architecture search. The aforementioned task embeddings cannot be directly applied to GNNs as the inputs do align across datasets. AUTOTRANSFER avoids the bottleneck by using asymptotic statistics of the Fisher Information Matrix with randomly initiated weights.
PROBLEM FORMULATION AND PRELIMINARIES
We first introduce formal definitions of data structures relevant AUTOTRANSFER.
3

---

Published as a conference paper at ICLR 2023
Rand Inits
FIM
Scale-Invariant Measure
Training Objective
Model
a2
Task
Task
Task Fealure
Task Eimbedding
z
Ze
M,
Q2
g
Margin Ranking
Model
To efficiently embed task, first extract task features by concatenating features measured from R randomly initialized anchor models. Then, we introduce projection function g()) with learned weights to transform the task features into task embeddings. Right: Training objective for optimizing g(-) with triplet supervision.
Figure 2: Pipeline for extracting task embeddings. Left:
Definition (Task) We denote task as T = (D, L()), consisting of a dataset 'D and a loss function E()) related to the evaluation metric.
For each training attempt on task (*),, we can record its model architecture M,. hyperparameters Hj. and corresponding value of loss lj. i.e., (M,, HM. 1z). We propose maintain a task-mode) bank to facilitate knowledge transfer to future novel tasks.
Definition (Task-Model Bank) A task-model bank B is defined as a collection of tasks, each with multiple training attempts, in the form of B = {(T()) ((M)) .HS (1)))
AutoML with Knowledge Transfer. Suppose we have a task-model bank B. Given novel task T(a) which has not been seen before, our goal to quickly find model that works reasonably well on the novel task by utilizing knowledge from the task task-model bank.
In this paper, we focus on AutoML for graph learning tasks, though our developed technique is general and can be applied to other domains. We define the input graph G {V, E}, where V the node set and F. C V x V is the edge set. Furthermore, let denote its output labels, which can be node-level, edge-level graph-level. A GNN parameterized by weights 0 outputs posterior distribution P(G, y, 0) for label predictions.
4 PROPOSED SOLUTION: AUTOTRANSFER
In this section, introduce the proposed RANSSER solution. AUTOTRANSFER uses the task embedding space as tool to understand the relevance of previous architectural designs to the target task. The designed task embedding captures the performance rankings of different architectures different tasks while also being efficient to compute. We first introduce theoretically motivated solution extract scale-invariant performance representation of each task-model pair. We use these representations to construct task features and further learn task embeddings. These embeddings form the task embedding space that we finally use during the AutoML search.
4.1
BASICS OF THE FISHER INFORMATION MATRIX (FIM)
Given a GNN defined above, its Fisher Information Matrix (FIM) F is defined as
F Eg.y, [ V log P(G, y, 0) V log y)
which formally is the expected covariance of the scores with respect to the model parameters. There two popular geometric views for the FIM. First, the FIM is upper bound of the Hessian and coincides with the Hessian the gradient is O. Thus, the FIM characterizes the local landscape of the loss function near the global minimum. Second, similar to the Hessian, the FIM models the loss
4

---

Published as a conference paper at ICLR 2023
landscape with respect not to the input space, but to the parameter space. In the information geometry view, if we add small perturbation to the parameter space, we have
(L(((G, y. ()I y, 8 + d0)) = d0 Fd8.
where (-(,,)) stands for Kullback-eibblrr divergence. means that the parameter space of a model forms& Riemannian manifold and the FIM works as its Riemannian metric. The FIM thus allows us to quantify the importance of model's weights in a way that is applicable to different architectures.
4.2 FIM-BASED TASK FEATURES
Scale-ivvariant Representation of Task-Model Pairs. We aim to find a scale-invariant represen- tation for each task-model pair which will form the basis for constructing task features. The major challenge in using the FIM to represent GNN performance is that graph datasets do not have a universal, fixed input structure, it is infeasible find single pre-trained model and extract its FIM. However, training multiple networks poses problem as the FIMs computed for different networks are not directly comparable. We choose to use multiple networks but additionally propose to use asymptotic statistics of the FIM associated with randomly initialized weights. The theoretical justification for the relationship between the asymptotic statistics of the FIM and the trainability of neural networks was studied in (Karakida et al. 2019 Pennington and Worah 2018) to which we refer the readers. We hypothesize that such a measure of trainability encodes loss landscapes and generalization ability and thus correlates with final model performance the task. Another issue that relates to input structures of graph datasets that different models have different number of parameters. Despite some specially designed architectures, e.g., (Lee et al., 2019) Ma al.) 2019). most GNN architecture design can be represented as sequence of pre-processing layers, message passing layers, and post-processing layers. Pre-process layers and post-process layers are Multilayer Perceptron (MLP) layers, of which the dimensions vary across different tasks due to different input/output structures. Message passing layers are commonly regarded the key design for GNNs and the number of weight parameters can remain the same across tasks. In this light, we only consider the FIM with respect to the parameters in message passing layers so that the number of parameters considered stays the same for all datasets. We note that such formulation has its limitations, in the sense that it cannot cover all the GNN designs in the literature. We leave potential extensions with better coverage for future work. We further approximate the FIM by only considering the diagonal entries, which implicitly neglects the correlations between parameters. We note that this is common practice when analyzing the FIMs of deep neural networks, as the full FIM is massive (quadratic in the number of parameters) and infeasible to compute even on modern hardware. Similar to Pennington and Worah (018), we consider the first moments of FIM
m1 = tr [ and m2
F2 n
and use m2/mī as the scale-invariant representation. The computed is lower bounded by I and captures how concentrated the spectrum is. A small a indicates the loss landscape flat, and its corresponding model design enjoys fast first-order optimization and potentially better generalization. To encode label space information into each task, we propose to train only the last linear layer of each model given task, which can be done efficiently. The parameters in other layers frozen after being randomly initialized. We take the average over R initializations to estimate the average â.
Constructing Task Features. We denote task features as measures extracted from each task that characterize its important traits. The design of task features should reflect our final objective: to use these features to identify similar tasks and transfer the best design distributions. Thus, we select U model designs as anchor models and concatenate the scale-invariant representations ā, of each design as task features. To retain only the relative ranking among anchor model designs, we normalize the concatenated feature vector to a scale of 1. We let z denote the normalized task feature.
4.3 FROM TASK FEATURES TO TASK EMBEDDINGS
The task feature zy introduced above can be regarded means of feature engineering. We construct the feature vector with domain knowledge, but there is no guarantee it functions as anticipated. We thus propose to learn projection function g(.) : TR IR that maps task feature 2, final task
5

---

Published as a conference paper at ICLR 2023
embedding 2, = =(Zz). We do not have any pointwise supervision that can be used as the training objective. Instead, we consider the metric space defined by GraphGym. The distance function in GraphGym computed using the Kendall rank correlation between performance rankings of anchor models trained on the two compared tasks correlates nicely with our desired knowledge transfer objective. It is not meaningful to enforce that task embeddings mimic GraphCimm's exact metric space, as GraphGym's 's metric space can still contain noise, or does not fully align with the transfer objective. We consider surrogate loss that enforces only the rank order among tasks, To illustrate,
let us consider tasks ( T(k) and their corresponding task emheddings. = ze Ze Note
that ze is normalized to 1 so (1) Ze measures the cosine similarity between tasks T()) and T())
(i)
Let dg(: -) denote the distance estimated by GraphGym. We want to enforee
(6)
(k) if (CTT,,T0) d, (1T
To achieve this, we use the margin ranking loss surrogate supervised objective function:
_= z(6) = ,)) = max(0, y (z! _e
(1)
(k) + margin).
(2)
Here if (T()), TW))= < d,,Tio ) then we have its corresponding label y = 1, and y/
otherwise. Our final task embedding space is then FIM-based metric space with cosine distance
function, where the distance is defined as de (T((1, TÚ)) =!= (i) Please refer to the detailed
training pipeline at Algorithm 2 in the Appendix.
4.4 AUTOML SEARCH ALGORITHM WITH TASK EMBEDDINGS
To transfer knowledge to a novel task, a naive idea would be to directly carry over the best model configuration from the closest task in the bank. However, even a high Kendall rank correlation between model performance rankings of two tasks 7(*), 7()) does not guarantee the best model configuration in task ( will also achieve the best performance on task TW), In addition, since task similarities subject to noise, this naive solution may struggle when there exist multiple reference tasks that are all highly similar.
To make the knowledge transfer more robust to such failure cases, we introduce the notion of design distributions that depend on top performing model designs and propose to transfer design distributions rather than the best design configurations. Formally, consider a task ( in the task- model bank B, associated with its trials ((M f7(") ...0)). We can summarize its designs as a list of configurations C' {CI...... cw), }. such that all potential combinations of model architectures M and hyperparameters H fall under the Cartesian product of the configurations. For example, 01 could be the instantiation of aggregation layers, and 02 could be the start learning rate. We then define design distributions as random variables C₁ C2..... CW, each corresponding to a hyperparameter. Each random variable e is defined as the frequency distribution of the design choices used in the top K trials. We multiply all distributions for the individual configurations .c......ww to approximate the overall task's design distribution ()) = 1I... P(c.. (+))..
During inference, given a novel task T(a), we select a close task subset S by thresholding task embedding distances, i.e., S [d. (T) ( < detres)) We then derive the transferred design prior (C) of the novel task by weighting design distributions from the close task subset S.
P((C)(())) ErwEs (T1) P(C
Eroes (_(()),T())
The inferred design prior for the novel task can then be used to guide various search algorithms. The most natural choice for few trial regime is random search. Rather than sampling each design configuration following a uniform distribution, we propose to sample from the task-informed design prior (C)T(()) Please refer to Appendix A to check how we augment other search algorithms.
For AUTOTRANSFER, we {(D(), ((),(((())))))) 2. "0 P(CTT()) as our pipeline only requires using task embedding z and design distribution (()) rather than detailed training trials. detailed search pipeline is summarized in Algorithm
preprocess the task-model bank B into B,

---

Published as a conference paper at ICLR 2023
Algorithm 1 Summary of AUTOTRANSFER search pipeline
Require: processed task-model bank B, = {(D(3), ()), 2:2 P(CIT())), novel task
T(n) = (D(x), (()) U anchor models M1,.., Mr, H specifies the number of repeats.
1: for 1 to U do
2:
1 to R do 3: Initialize weights 0 for anchor model M,, randomly
4: Estimate FIM F - Ep Vo log (Ma 0) Vo log P(Mu, 0)
5: Extract scale-invarian representation a m2/m/ /m/ following Eq.
6: end for
7: Ou <- mean (i au
(1)
)
8: end for
9:
+ concat( (1 ag...., av)
(n) + -g(2)")
ze
Select close task subset S [T (1)) 2100 dihres}
Get design prior (C(C(((()) by aggregating subset S following Eq.
3
Start HPO search algorithm with the task-informed design prior P, (C(C(())
EXPERIMENTS
5.1 EXPERIMENTAL. SETUP
Task-Model Bank: GNN-BANK-101.
To facilitate AutoML research with knowledge transfer, collected GNN-BANK- 101 the first large-scale graph database that records reproducible design configurations and detailed training performance variety of tasks. Specifically, GNN- MENN-ANK-100 currently includes six tasks for
node classification (AmazonComputee) (Shchur et al.) 2018). AmazonPhoto (Shchur et al.
2018).
CiteScer (Yang al. 2016). CoauthorCS (Shchur et al. 2018). CoauthorPhysies (Shchur
2018
Cora (Yang et al.] (016)) and six tasks for graph classification (PROTEINS (Taaovv et al.
2019).
BZR (vanov et al. 2019), COX2 (Ivanov et al., 2019). DD VIvanov al., 2019 ENZYMES Ivanov
ct al. 2019). IMBB-M (vanov et (019)). Our design space follows Youu et al. 2020), and we extend the design space include various commonly adopted graph convolution and activation layers. We extensively run 10,000 different models for each task, leading to 120,000 total task-model combinations, and record all training information including train/val/test loss.
Benchmark Datasets. We benchmark AUTOTRANSFER on six different datasets following prior work (Qin et al. 2021). ). Our datasets include three standard node classification datasets (Coauthor- Physics (Shchur et al. 2018), CoraFull (Bojchevski and Günnemann 2017) and OGB-Arxiv (Hu et al. 2020)), well as three standard benchmark graph classification datasets, (COX2 Avanov al. 2019
IMDB-M (Ivanov et al. 2019) and PROTEINS (Ivanov et al. 2019). CoauthorPhysics and CoraFull transductive node c classification datasets, so randomly assign nodes into train/valid/eest sets fol- lowing 50%:25%:22 225% split (Qin et al. 2021). We randomly split graphs following a 80%: 10% split for the three graph classification datasets QQn al. 2021). We follow the default train/salid/test split for the OGB-Arxiv dataset (Hu et al. 2020). To make sure there is no information leakage, we temporarily remove all records related to the task from our moskmodee bank if the dataset we benchmark was collected in the task-model bank.
Baselines. We compare our methods with the state-of-the-ard approaches for GNN AutoMI. We use GCN and GAT with default architectures following their original implementation as baselines. For multi-trial NAS methods,
consider GraphNAS (Gao et al. 2020). For one-shot NAS methods, include DARTS (Liu et al. 2018) and GASSO (Qin et al., (021). GASSO is designed for transductive settings, so omit for graph classification benchmarks. We further provide results of HPO algorithms based on our proposed search space as baselines: Random, Evolution, TPE (Bergstra
et al. 2011) and HyperBand (Li et al. (017).
We by default allow searching 30 trials maximum for all the algorithms, i.e., an algorithm can train 30 different models and collect the model with the best accuracy. We use the default setting for one-shot
7

---

Published as a conference paper at ICLR 2023
Table 1: Performance comparisons of AUTOTRANSPER and other baselines. We report the average test accuracy and the standard deviation over ten runs. With only 3 trials AUTOTRANSFER already outperform most SOTA baselines with 30 trials.
Mode CoraFull
Graph IMDS
Mothod
Physics
OGB-Arxiv
CCX2
PROTEINS
GCN (30 trials)
95.88:10.16
67.12:10.52
70.46:10.18
29.23-2.119
50.40:1.3002
74.84-12.82
GAT (30 trials)
05.71=0..2
65.92-0068
68 08.32-00.32
81.56:44.17
49.672-4.30
75 3..30:33.72
GraphNAS (30 trials)
92.77=0.84 0.84
63.13=3322 28
65 65.90-22.64
71.73+1.40
46.93-23396
72.31+3.36
DARTS GASS
95.28:1 1.67
2.
69.02:11.18
79.82:1.115
50.26:1:.008
75.04:13..11
96.38
68.89
70.52
Random (3 trials)
95 16=
61 2.24444444
67 67.92-t1 192
76.88+33. 17
45.79 4.39
72.47+22.57
TPF (30 trials)
96.41 36
06.37-11.7
71.35::0.44
82.27:4..01
50.33-1-.000
79.4611.23
HyperBand (30 trials)
96.56=0.30
67.75#1.21 121
71.60+00.36
82.21=11799
50.86+3.445
79.32+1 1.16
AUTOTRANSFER trials) AUTOTRANSFER (30 trials)
96.64-0042 96.91
09.27-0.777
71:42:00
1.59
52.33++2.13
77.81-2.19
70.05-10:22
72.21:10:27
86.52.11.55
$4.93-11.25
81.25
NAS algorithms (DARTS and GASSO), as they only train super-model once and can efficiently evaluate different architectures. We are mostly interested in studying the few-trial regime where most advanced search algorithms degrade to random search. Thus we additionally include random search (3 trials) baseline where we pick the best model out of only trials.
5.2 EXPERIMENTS ON SEARCH EFFICIENCY
We evaluate AUTOTRANSFER by reporting the average best test accuracy among all trials considered over ten runs of each algorithm in Table The test accuracy collected for each trial is selected at the epoch with the best validation accuracy. By comparing results from random search (3 trials) and AUTOTRANSFER (3 trials), we show that our transferred task-informed design prior significantly improves test accuracy in the few-trial regime, and can be very useful environments that are where computationally constrained. Even if we increase the number of search trials to 30, AUTOTRANSFER still demonstrates non-trivial improvement compared with TPE, indicating that our proposed pipeline has advantages even when computational resources are abundant. Notably, with only 3 search trials, AUTOTRANSFER surpasses most of the baselines, even those that use 30 trials.
To better understand the sample efficiency of AUTOTRANSFER, plot the best test accuracy found at each trial in F for OGB-Arxi and TU-PROTEINS datasets. We notice that the advanced search algorithms (Evolution and TPE) have no advantages over random search at the few-trial regime since the amount of prior search data is not yet sufficient to infer potentially better design configurations, On the contrary, by sampling from the transferred design prior, AUTOTRANSFER achieves significantly better average test accuracy in the first few trials. The best test accuracy at trial 3 of AUTOTRANSFER surpasses its counterpart at trial 10 for every other method.
Search Curves on OGB-Andv
Search Curves on TU-PROTEINS
Teal Aoc
Method
Method
Random
Random
TPE
TPE AutoTranster
AutoTranstor
4
10
2
10
Triate
Trialk
Performance comparisons in the few-trial regime. At trial f, we plot the best test accuracy among all models searched from trial 1 to trial f. AUTOTRANSFER can reduce the number of trials needed to search by an order of magnitude (see also Table in Appendix).
Figure 3:
5.3 ANALYSIS OF TASK EMBEDDINGS
Qualitative analysis of task features. To examine the quality of the proposed task features, we visualize the proposed task similarity matrix (Figure 4 (b)) along with the task similarity matrix (Figure (a)) proposed in GraphGim. We show that our proposed task similarity matrix captures similar patterns GraphGym's task similarity matrix while being computed much more efficiently
Results come from the original paper Qin al. 2021
8

---

Published as a conference paper at ICLR 2023
GraphSym Task artty
Proposed Task Similartt
Comditiion Foevdrng of TTee
Aratenddrppp putark rode
Oxdergede
1.3
Texs Enbeddnn
Conunde
-34
Co DOL
TOLROTTHO poe
TU BOLOOG oren
Tu Do pap
TOLMOS RULMOL
Ti Co0 aa So To TUJM0OSP Sre
m gnd
ErSeev one
(b)
Figure 4: (a) GraphGym's task similarity between all pairs of tasks (computed from the Kendall rank correlation between performance rankings of models trained on the two compared tasks), higher value represents a higher similarity. (b) The proposed task similarity computed by computing the dot-proe dot-product between extracted task features. (c) The Kendall rank correlation of similarity rankings of the other tasks with respect to the central task between the proposed method and GraphGym.
by omitting training. We notice that the same type of tasks, i.e., node classification and graph classification, share more similarities within each group. As a sanity check, we examined that the closest task in the bank with respect to CoraFull is Cora. The top 3 closest tasks for OGB-Arxiv are AmazonComputers, AmazonPhoto, and CoauthorPhysics,
of which are node classification tasks.
Generalization of projection function g(-). To show the proposed projection function g(.) can generate task embeddings that can generalize to novel tasks, conduct leave-one-out cross validation with all tasks in our task-model bank. Concretely, for each task considered as novel task T(",, we use the rest of the tasks, along with their distance metric d, (---) estimated by GraphGym's 's exact but computationally expensive metric space, to train the projection function g(+). We calculate Kendall rank correlation over task similarities for Task Feature (without g(-)) and Task Embedding (with g(-)) against the exact task similarities. The average rank correlation and the standard deviation over ten runs is shown on Figure 4 (c). We find that with the proposed g(-). our task embeddings indeed correlate better with the exact task similarities, and therefore, generalize better novel tasks.
Ablation study alternative task space design. To demonstrate the superiority of the proposed task embedding, we further compare it with alternative task features. Following prior work (Yang let al. 2019),, use the normalized losses over the first 10 steps as the task feature. The results on OGB--Arivv are shown in Table 2 Compared to AUTOTRANSFER
task embedding, the task feature induced by normalized losses has lower ranking correlation with the exact metric and yields worse performance. Table 2 further justifies the efficacy of using the Kendall rank correlation as the metric for task embedding quality, as higher Kendall rank correlation leads to better performance.
Table 2: Ablation study on the alternative task space design versus AUTOTRANSFER 's task embedding. We report the average test accuracy and the standard deviation OGB-Arxi over ten runs.
Kendall rank correlation
Test accuracy
Alternative: Normalized Loss AUTOTRANSFERR 's Task Feature AUTOTRANSFER'S Task Embedding
-0.07=0043
1.22
0. 0.18++0.30
70.67=0.52 71.42.10.39
0.43
CONCLUSION
In this paper, we study how to improve AutoML. search efficiency by transferring existing architectural design knowledge to novel tasks of interest. We introduce a task-model bank that captures the performance over diverse set of GNN architectures and tasks. We also introduce computationally efficient task embedding that can accurately measure the similarity between different tasks. We release ENN-BANK-1M 101, a large-scale database that records detailed GNN training information of 120,000 task-model combinations. We hope this work can facilitate and inspire future research in efficient Al to make deep learning more accessible general audience.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Xiang Lisa Li, Hongyu Ren, Yingxin Wu for discussions and for providing feedback on our manuscript. We also gratefully acknowledge the support of DARPA under Nos. HR00I 12100099 (TAMI,, N66001 I 1924033 (MCS); ARO under Nos. W91 LANN-16-1-0344 (MURI), W91 INT-16-1- 0171 (DURIP); NSF under Nos. OAC- (CINES), OAC-1934578 (HDR), CCF-191 1940 (Expeditions), NIH under No. 3U54HG010426 04S1 (HuBMAP), Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks, KDDI, NEC, and Toshiba. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-lcarning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6430-6439, 2019.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011.
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv: 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1 2020.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332. 2018.
Shaofei Cai, Liang Li, Jincan Deng, Beichen Zhang, Zheng-Jun Zha, Li Su, and Qingming Huang, Re- thinking graph neural architecture search from message-ppassngg In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 6657-6666, 2021.
Yuhui Ding, Quanming Yao. Huan Zhao, and Tong Zhang. Diffig: Differentiable meta graph search for heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 279 288, 2021.
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabulaa Robust and accurate automl for structured data. ICML 2020 Workshop on Automated Machine Learning, 2020.
Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graphnas: Graph neural architecture search with reinforcement learning, arXiv preprint arXiv: 1e011994499999 2019.
Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graph neural architecture search. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 1403-1409 International Joint Conferences on Artificial Intelligence Organization, 7 2020. URL http::://ool.oorg//.222///////e /020/
195
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference Computer Vision, pages 560. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Xiangnan He, Kuan Deng, Xiang Wang. Yan Li, Yongdong Zhang, and Meng Wang. Lightenn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 639 648, 2020.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV), pages 784-800, 2018.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems,
2020.

---

Published as a conference paper at ICLR 2023
Sergei Ivanov, Sergei Sviridov, and Evgeny Burnaev. Understanding isomorphism bias in graph data sets, 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:/11.0984/ 2017.
Wonyong Jeong, Hayeon Lee, Geon Park, Eunyoung Hyung, Jinheon Baek, and Sung Ju Hwang. Task- adaptive neural network search with meta-contrastive learning. Advances in Neural Information Processing Systems, 34:213
2021.
Donald R Jones. A taxonomy of global optimization methods based response surfaces. Journal of global optimization, 21(4)3345-383, 2001.
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. The 22nd International Conference on Artificial Intelligence and Statistics, pages 1032-1041. PMLR, 2019.
Cat P Le, Mohammadreza Soltani, Juncheng Dong, and Vahid Tarokh. Fisher task distance and its application in neural architecture search. IEEE Access, 10:47235-47249, 2022.
Erin LeDell and Sebastien Poirier. H2o automl: Scalable automatic machine learning. ICML 2020 Workshop on Automated Machine Learning. 2020.
Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In International coneraationalcooffeenncc conference on machine learning, pages 3734-3743, PMLR, 2019.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit- approach to hyperparameter optimization. The Journal of Machine Learning Research, 18()):676566866 2017.
Yanxi Li, Zean Wen, Yunhe Wang, and Chang Xu. One-shot graph neural architecture search with dynamic search space. In Proc. AAAI Conf. Artif. Intell, volume 35, pages 8510-8517, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Diffcrentiable architecture search. In International Conference on Learning Representations, 2018.
Yao Ma, Suhang Wang, Charu C Aggarwal, and Jiliang Tang. Graph convolutional networks with eigenpooling. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 723-731, 2019.
Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of single- hidden-layer neural network. Advances neural information processing systems, 31, 2018.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez- Sanchez-Gonzale Gonzalez, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference Learning Representations, 2020.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095-4104 4104. PMLR, 2018
Yijian Qin, Xin Wang, Zeyang Zhang, and Wenwu Zhu. Graph differentiable architecture search with structure learning. Advances in Neural Information Processing Systems, 34, 2021.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe V
Lc, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning, pages 2902-2911 I. PMLR, 2017.
Roseanna W Saaty. The analytic hierarchy process- what it is and how it is used. Mathernatiaa modelling, 93-5):161--776, 1987.
Alvaro Gonzalez. Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec. and Peter Battaglia. Learning to simulate complex physics with graph networks. International Conference Machine Learning, pages 8468. PMLR, 2020. 2020.
12

---

Published as a conference paper at ICLR 2023
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: 181.0888 2018.
Lanning Wei, Huan Zhao, Quanming Yao, and Zhiqiang He. Pooling architecture search for graph classification. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2091-2100, 2021.
Catherine Wong, Neil Houlsby, Yifeng Lu, and Andrea Gesmundo. Transfer learning with neural automl. Advances in neural information processing systems, 31, 2018.
Chengrun Yang. Yuji Akimoto, Dac Won Kim, and Madeleine Udell. Oboe: Collaborative filtering
for automl model selection. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery data mining, 2019.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48. PMLR, 2016.
Rex Ying. Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pages 974- 983, 2018.
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33: 1300911117,,, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-122 12132. PMLR, 2021.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3712-3722, 2018.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference Machine Learning, pages 12674-12683 PMLR, 2021.
Huan Zhao, Quanming Yao, and Weiwei Tu. Search to aggregate neighborhood for graph neural network. arXiv preprint arXiv:2104.06600 2021.
Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. Auto-gnn: Neural architecture search of graph neural networks. arXiv preprint arXiv: 19090039999444 2019.
Marinka Zitnik, Monica Agrawal, and Jure Leskovec. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13))1457-i46) 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1rXiv:1611.01578 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018.
13

---

Published as a conference paper at ICLR 2023
A ADDITIONAL IMPLEMENTATION DETAILS
Runtime analysis. We have empirically shown that AUTOTRANSFER can significantly improve search efficiency by reducing the number of trials needed to achieve reasonably good accuracy. The only overhead we introduced is the procedure of estimating task embeddings. Since we use randomly initialized architecture, extracting each task feature only requires at most one forward and a few backward passes from single minibatch of data. The wall-clock time depends the size of the network and data structure. In our experiments, it typically takes a few seconds on an NVIDIA T4 GPU. We repeat the task feature extraction process 5 times for each anchor model, and for total of 12 anchor models. Thus, the wall-clock time of the overhead for computing task embedding of novel task is within few minutes. We note the length of this process is generally comparable to one trial of training on a small-sized dataset, and the time saved is much more significant for large-scale datasets.
Details for task-model-bank training. Our GNN model specifications are summarized in Table 3 Our code base was developed based on GraphGym YYou et al., 2020p. For all the training trials, We use the Adam optimizer and cosine learning rate scheduler (anncaled to O, no restarting). We use L2 regularization with weight decay of 5e-4. We record losses and accuracies for training, validation and test splits every 20 epochs.
Training details for AUTOTRANSFER We summarize the training procedure for the projection function g(-) in Algorithm 2 We set U = 12 and R = 5 throughout the paper. We use the same set of anchor model designs as those in GraphGym. We use two-layer MLP with a hidden dimension of 16 to parameterize the projection function g()). We use the Adam optimizer with a learning rate of 5e-3. We margin 0.1 and train the network for 1000 iterations with batch size of 128. We adopt K = 16 when selecting the top trials that are used to summarize design distributions.
Details for adapting TPE, evolution algorithms. We hereby illustrate how to compose the search algorithms with the transferred design priors. TPE is Bayesian hyperparameter optimization method, meaning that it is initialized with a prior distribution to model the search space and updates the prior as it evaluates hyperparemeter configurations and records their performance. We replace this prior distribution with the lask-informed design priors. As evolutionary algorithms generally initialize a large population and iteratively prune and mutate existing networks, we replace the random network initialization with the task-informed design priors. As we mainly focus on the few trial search regime, we set the fully random warm-up trials for both TPE and evolution algorithms.
Table 3: Design choices in our search space
Type
Choices
Convolution
GeneralConv, WWWCCCO/SAAGCCCO,,GGNCCOO 1, 4 Sum, Mean-Pooling, Max-Pooling ReLU, PReLU, leaky_ReLU,, ReLU, ELU 64, 256
Number of heads Aggregation
Activation
Hidden dimension
Layer
connectivity
Stack, Skip-Sum, Skip-Concat 1.2
Pre-proe Pre-process layers
Message passing layers
2,4.6.8
Post-process layers
2, 3
Learning rate
0. 0.001
Training epochs
200, 800, 1600
B ADDITIONAL DISCUSSION
Limitations. In principle, AUTOTRANSFER leverages the correlation between model performance rankings among tasks to efficiently construct model priors. Thus, it is less effective if the novel task has large task distances with respect to all tasks in the task moskl bank. In practice, users can continuously add additional search trials to the bank. As the bank size grows, it will be less likely that novel task has a low correlation with all tasks in the bank.
14

---

Published as a conference paper at ICLR 2023
Algorithm 2 Training Pipeline for the projection function g(-)
Require: Task features (2())) extracted for each task from the task-model bank. Distance
measure dg(:,-) estimated in GraphGym.
for each iteration do
2:
Sample T(1) _
(k)
g( (1)
(() g((()))))
3:
ze
20
Ze
4: y+ 1 if (T(), (() dg(T()) T(()) else -1
5: Optimize objective function C, (z) 2e Zc 9 in Eq.
(b)
6: end for
Social Impact. Our long-term goal is to provide a seamless GNN infrastructure that simplifies the training and deployment of ML models structured data. Efficient and robust AutoML algorithms are crucial to making deep learning more accessible to people who are interested but lack deep learning expertise well as those who lack the massive computational budget AutoML traditionally requires. We believe this paper is an important step to provide AI tools to a broader population and thus allow for Al help enhance human productivity. The datasets used for experiments among the most widely-used benchmarks, which should not contain any undesirable bias. Besides, training/test losses and accuracies are highly summarized statisties that we believe should not incur potential privacy issues.
C ADDITIONAL RESULTS
Search efficiency. We summarize the average number of trials needed to surpass the average best accuracy found by TPE with 30 trials in Table We show that AUTOTRANSER reduces the number of explored architectures by an order of magnitude.
Table 4: Average number search trials needed surpass the average best result found by TPE with 30 trials
Node Corafull
Graph IMDB
Physics
OGB-Arxiv
CoX2
PROTEINS
Num. of Trials Accuracy
2 6..85
3 71.42+0.39
6 80.21
96.64=0.42
82.96 + 1.75
52.33++2.13
Ablation study on number of anchor models and task embedding design. We empirically demonstrate how the number of anchor models affect the rank correlation in Table 5 While 3 anchor models are not enough to capture the task distance, we found that and 12 have a satisfactory empirically trade-off between capturing task distance and computational efficiency. Furthermore, in Table Table([]demonstrate demonstrate that the learned task embedding space is superior to the proposed task feature space terms of correlation as well as final search performance.
Table 5: Average Kendall rank correlation of similarity rankings of the other tasks with respect the central task between the proposed method and GraphGym.
Num. of anchor models
6
12
Task Feature Task Embedding
0.03+10.34
0.11
0.16::0.4
0.18::0030
0.12-00.28
0.26++.30
0.36+0.22
0.43-+-0.22
Visualizing model designs. We visualized the transferred design distributions and ground truth design distributions the TU-PROTEINS dataset in Figure 5 well as Coauthor-Physe Physics dataset in Figure 6 We could observe that the transferred design distributions have positive correlation on most of the design choices.
15

---

Published as a conference paper at ICLR 2023
Table 6: Ablation study of the number of anchor models as well task embedding task feature on OGB-Arxiv with trials. We report the average test accuracy and the standard deviation over ten runs.
Num. of anchor models
3
12
Task Frature Task Embedding
69 0.82
69.86+0.78 70.59+0.63
70.41 0.59
70.67+00.52 71.42::0.39
69.80 0.75
71.16:0.44
2014
Figure 5: We plot the transferred design distributions and ground truth design distributions the TU-PROTEINS dataset.
Figure 6: We plot the transferred design distributions and ground truth design distributions the Coauthor-Physics dataset.
16


Reviews:
[{"review_id": "jnefw5jjZPd", "reviewer": "Reviewer_SdtT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"AutoTransfer: AutoML with Knowledge Transfer - An Application to Graph Neural Networks\" makes two major contributions: First, the authors devise a benchmark for searching graph neural networks, consisting of 12 task and comprising 120,000 evaluated task-model combinations. Second, they propose a method for transfer learning, suggesting a prior based on past experience for searching neural architectures on a new dataset.", "summary_of_the_review": "The paper makes good contributions to the field and provides some new artefacts for the community which can be quite useful to foster future research on NAS for GNNs. The paper has some minor issues which can however be fixed quite easily. Therefore, I would recommend to accept the paper under the condition that the issues are fixed for the camera ready version.", "strength_and_weaknesses": "[+] The provided benchmark dataset is valuable and hopefully facilitates and enables research on transfer learning graph neural networks.\n\n[+] The proposed transfer learning approach seems to perform reasonably well refining the state of the art.\n\n[+] To the best of my knowledge, the proposed method is novel.\n\n\n\n[-] The authors state that all GNNs can be described in terms of three stages: pre-processing with MLPs, message passing layers, and post-processing with MLPs. However, I would argue that this is quite restrictive and not able to reflect all possible architectures. See for example \n\nLee, Junhyun, Inyeop Lee, and Jaewoo Kang. \"Self-attention graph pooling.\" International conference on machine learning. PMLR, 2019.\n\nand \n\nMa, Yao, et al. \"Graph convolutional networks with eigenpooling.\" Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019.\n\nwhere pooling operations are added between the convolutions resulting in iterative changes of the graph structure. Of course, such a benchmark database needs to be restricted in some sense so that the set of candidates is reasonably limited. However, I would suggest to rephrase the statement in Section 4.2 a little bit so that it makes clear that the benchmark limits itself to such GNNs.\n\n[-] It is not clear why the authors chose the mentioned datasets for evaluation and why results on the other datasets are not reported.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VrN3KnCifKb", "reviewer": "Reviewer_1hga", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel AutoML method, AutoTransfer, for graph learning tasks. It introduces a task-model bank that can transfer the model design to similar new tasks. The experiments indicate that the method is efficient and improves the performance of the found models.", "summary_of_the_review": "The paper proposes an interesting method for transferring knowledge in AutoML for graph learning tasks.", "strength_and_weaknesses": "### Strengths\n1. The paper introduces an interesting method for transferring knowledge from a previously searched task to a new coming task with a task-model bank design.\n2. It proposes an interesting task embedding with several random-initialized anchor models, and the experiments show that the task similarity is promising.\n3. The methods are tested on multiple benchmarks and are proven to be efficient and performant.\n4. It releases a large task-model dataset that can inspire future research.\n\n### Weaknesses\n1. How are the anchor models selected? I failed to find the architecture design of the anchor models, and it would be good to see how diverse the anchor models should be.\n2. The graph architectures used in the paper are quite limited. For example, on the OGB leaderboard, many different GNN (non-GNN) algorithms are applied and achieve significantly better results. It would be interesting to have some of those models in the search space.\n3. It would be interesting to see some examples of the models selected for a group of tasks, to get some insights of the model design.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1qatitYiAP", "reviewer": "Reviewer_uQiP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studied NAS algorithm for GNN. The author proposed to transfer the prior architectural design knowledge to the novel task of interest. The author first extracts scale-invariant measures based on Fisher Information Matrix (FIM). The scale-invariant measures are used to form the task feature, which are further projected to task embeddings. The task embeddings are learned via a ranking loss built on top of GraphGym's extact metric space. Experiments justified the importance of the task embeddings in the AutoTransferer algorithm. The AutoTransferer also outperforms baselines like GraphNAS, \n", "summary_of_the_review": "Voted for weak accept because the algorithm is novel. However, I still have concerns pointed out in the Weakness section.\n", "strength_and_weaknesses": "Strength\n\n1. The idea of utilizing FIM to produce task-feature and further finetune it with ranking loss is novel.\n2. Ablation analysis justifies the effectiveness of the task embedding algorithm proposed by the paper.\n\n\nWeaknesses\n\n1. It seems that \"[AAAI2021] One-shot graph neural architecture search with dynamic search space\" also discussed task transfer. The author may need to compare with one-shot NAS or clarify the difference.\n2. The author pointed out that there are three stages in GNN architectures: 1) pre-processing, 2) message-passing, 3) post-processing. The author only considered message-passing layers and pointed out that it is commonly regarded as the most important part of GNN. However, since different tasks adopt different pre-/post-processing modules, it is not convincing to directly state tat message-passing is the most important.\n3. It will be good if the author can list the performance of SOTA algorithm in each dataset discussed in Table 1. This will help readers understand the gap between NAS models and human-engineered SOTA architectures.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster