Structuring Benchmark into Knowledge Graphs to Assist Large Language Models in Retrieving and Designing Models
Authors: Hanmo Liu
Shimin Di
Jialiang Wang
Zhili Wang
Jiachuan Wang
Xiaofang Zhou
Lei Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge Graph
Auto Machine Learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: In recent years, the design and transfer of neural network models have been widely studied due to their exceptional performance and capabilities. However, the complex nature of datasets and the vast architecture space pose significant challenges for both manual and automated algorithms in creating high-performance models. Inspired by researchers who design, train, and document the performance of various models across different datasets, this paper introduces a novel schema that transforms the benchmark data into a Knowledge Benchmark Graph (KBG), which primarily stores the facts in the form of performance(data, model). Constructing the KBG facilitates the structured storage of design knowledge, aiding subsequent model design and transfer. However, it is a non-trivial task to retrieve or design suitable neural networks based on the KBG, as real-world data are often off the records. To tackle this challenge, we propose transferring existing models stored in KBG by establishing correlations between unseen and previously seen datasets. Given that measuring dataset similarity is a complex and open-ended issue, we explore the potential for evaluating the correctness of the similarity function. Then, we further integrate the KBG with Large Language Models (LLMs), assisting LLMs to think and retrieve existing model knowledge in a manner akin to humans when designing or transferring models. We demonstrate our method specifically in the context of Graph Neural Network (GNN) architecture design, constructing a KBG (with 26,206 models, 211,669 performance records, and 2,540,064 facts) and validating the effectiveness of leveraging the KBG to promote GNN architecture design.

Published as a conference paper at ICLR 2025
STRUCTURING BENCHMARK INTO KNOWLEDGE GRAPHS TO ASSIST LARGE LANGUAGE MODELS IN RETRIEVING AND DESIGNING MODELS
Hanmo Liu
Shimin Di Jialiang Wang'
Zhili Wangla
Jiachuan Wang' Xiaofang Zhou' Lei Chen
'HKUST HKKUST (GZ) 'Southeast University
fhliubm, sdiaa, jwangic, zwangeo, jwangey) econnect.. ust.hh (2xf, leichen @usht @ust.hk
ABSTRACT
In recent years, the design and transfer of neural network models have been widely studied due to their exceptional performance and capabilities. However, the com- plex nature of datasets and the vast architecture space pose significant challenges for both manual and automated algorithms in creating high-performance mod- els. Inspired by researchers who design, train, and document the performance of various models across different datasets, this paper introduces novel schema that transforms the benchmark data into a Knowledge Benchmark Graph (KBG), which primarily stores the facts in the form of per formance data, model). Con- structing the KBG facilitates the structured storage of design knowledge, aiding subsequent model design and transfer. However, is non-trivial task to retrieve or design suitable neural networks based on the KBG, as real-world data are often off the records. To tackle this challenge, we propose transferring existing mod- els stored in KBG by establishing correlations between unseen and previously seen datasets. Given that measuring dataset similarity ended issue, we explore the potential for evaluating the correctness of the simi- larity function. Then, we further integrate the KBG with Large Language Models (LLMss,, assisting LLMs think and retrieve existing model knowledge in man- ner akin to humans when designing or transferring models. We demonstrate our method specifically in the context of Graph Neural Network (GNN) architecture design, constructing a KBG (with 26,206 models, 21 211,669 performance records and 2,540,064 facts) and validating the effectiveness of leveraging the KBG promote GNN architecture design.
complex and open-
INTRODUCTION
Designing neural networks (NNs) has traditionally been a complex and iterative process that relies heavily on human expertise and extensive experimentation (He et al. 2021). Manually crafting ar- chitectures requires deep understanding of both the data and the underlying algorithms, posing significant challenges when dealing with diverse and intricate datasets. To alleviate this burden, tomated methods have been developed to explore the vast architecture space more efficiently (Li & Talwalkar 2019 Real et al. 2019 Zoph & Le. 2017). Recently, Large Language Models (LLMs) like GPT-4 have emerged as powerful tools that can assist in neural network design (Tornede et al., 2024 Leveraging their expansive knowledge and reasoning capabilitiess LLMs can generate archi-
tectural suggestions and configuration for various types of neural networks, such as Convolutional
Neural Networks (Zheng et al. [2023 Yu al, 2023). Recurrent Neural Networks (Guo al. 2024 and Graph Neural Networks (GNNs) (Wang et al.) 2023al Wei et al. 2023) Wang et al. 2024). of-
fering new avenues for automating and enhancing the model design process.
Despite these advancements, designing NN architectures for unseen datasets remains time- consuming and demands iterative fine-tuning configurations achieve optimal performance on
"Corresooniigg author

---

Published as a conference paper at ICLR 2025
Benchmark Data
Knowledge Benchmark Graph
Dataset
Model
Acc
Seen Data
Acc
Model
D
Relevence?
D
of
Fi
91%
91%
F
of
82%
KBG Extraction
Dj
82%
Retrieve
Semantics Statistics
Semontics Statistics
84%
F1
84%
75%
D2
75%
86%
86%
09
86%
D1
F3
D2
F2
Performance?
Unseen Data D
D
Figure 1: An illustration of constructing Knowledge Benchmark Graph and retrieving models.
specific tasks. This intricate procedure necessitates a profound understanding of both theoretical concepts and practical data considerations. Existing automated algorithms, such as traditional neural architecture search methods (He et al. [021), aim to streamline this process but often do not lever- age prior knowledge, frequently starting from scratch without considering existing design patterns or performance insights. Moreover, while L.LMs have shown promise suggesting NN architectures (Wang ct al. 2023a| Dong et al. 2023), they tend to produce commonplace designs and may lack meticulousness in hyperparameter tuning <Wang et al. 2024). These limitations arise because they do not possess the detailed, structured knowledge necessary for customized model design tailored to specific datasets and tasks.
Recognizing these challenges, observe that wealth of design knowledge already exists in the form of benchmark data (Chitty-Venkata Venkata et al.) 2023 Wang et al. 2024 Ying et al. (019b)), where scholars and experts have documented the performance of various models across different datasets. However, this valuable resource has not been effectively leveraged. To bridge this gap. we propose a novel schema that transforms benchmark data into Knowledge Benchmark Graph (KBG) capable information. As shown in Fig. entities in the KBG may
of storing the include datasets and models, which can be further annotated with statistical properties, semantic descriptions, hyperparameter settings (see more details in Fig, 2 Then, the facts in KBG can capture the performance between data and models, including loss, accuracy, etc. This structured knowledge facilitates the retrieval and utilization of prior design insights, thereby aiding subsequent model design and transfer.
Intuitively, with KBG that describes the tonta--mdel--erformmaee rithms only need to retrieve a well-performing model for a given dataset. However, utilizing this KBG design an NN model is non-trivial. The KBG is constructed based on observed data and existing models, yet the data encountered by users are frequently unseen before (i.e., not included in the KBG). Consequently, establishing the relevance between datasets becomes critical challenge that must be addressed. To tackle this challenge, we establish interconnections between datasets and models within the KBG. We evaluate the unsupervised relevance among datasets based on their statistical properties and semantic descriptions. This approach allows us to infer relationships and complete the KBG for datasets lacking direct performance records. However, is worth noting that how to construct similarity between datasets is still an open and challenging topic. Not only should we consider the similarity of the data structure, domain, description, and statistics, but we also need to consider whether one model performs similarly on the two datasets. To aid future work data similarity in model design, we propose a novel metric that prioritizes relevant insights effectively.
relationship, automated algo-
To validate that the above KBG idea can promote model design, we subsequently propose to in- tegrate L.LSs with the KBG to assist L.MS in designing the NN models. This integration enables automated algorithms to think and retrieve existing model knowledge in manner akin to human ex- perts when designing or transferring models. We demonstrate this process specifically in the context of Graph Neural Network architecture design, validating the potential of constructing and leveraging the benchmark KG to enhance LLM--riven network design. Our contributions are summarized as:
We introduce a novel schema that transforms benchmark data into a Knowledge Benchmark Graph, summarizing the formation such as data statistics, model architecture, and model hyperparameters, which facilitate the structured storage of design knowledge aid subsequent model design and transfer.
relationships. This KBG can also record in-
To handle unseen data, we propose to model the relevance between datasets derived from inherent data characteristics. Furthermore, we also develop a novel evaluation metric to discuss how well the relevance scores prioritize useful insights.
2

---

Published as a conference paper at ICLR 2025
We validate the KBG idea in the context of GNN architecture design, where construct graph with 26,206 models, 21 1,669 performance records and 2,540,064 facts. Besides, we integrate the KBG with LLMs to promote the performance and efficiency of designing neural networks.
RELATED WORKS OF AUTOMATED MACHINE LEARNING
Automated Machine Learning (AutoML) has emerged critical approach automate the design and optimization of machine learning models. Traditional AutoML. methods aim to streamline the model development process by automatically processing data (Nargesian et al. 2017 Khurana al. 2017). performing neural architecture search (NAS) (Ren et al. 2020), and tuning hyperparameters (Shahriari al. 2016) without requiring extensive human expertise. The goal of neural network design is to identify the best- best-performing model F for an unseen dataset D from large space F;
M(F: D*) = max M(F; D"), FEF
(1)
where MF; D*) evaluates how model F performs on dataset D This automation reduces the barriers for non-experts while enhancing the efficiency of model development for experts. Despite its potential, traditional AutoML faces significant challenges, particularly in its scalability and ability to generalize across wide variety of datasets and tasks. On the one hand, the search space for neural architectures and hyperparameters is vast, making it computationally expensive to explore all possible configurations (typically several GPU hours days) (Salchin al.) [023 Liu al., 2022), even accelerated by AutoML algorithms. On the other hand, the performance predictions of AutoML models are often limited to the datasets they trained on, leading suboptimal generalization to unseen data (Wen et al., 2019 Zheng et al., 2020).
Recently, to address the efficiency issue, there is a line of training-free AutoML methods (Mellor ct al., 2020 Lopes et al., (202T) that estimates the suggested model performance by using a sin- gle forward or backward computation
a single minibatch of data without full-training (Tanaka et al. 2020; Xing et al., [2024). They design various score functions to evaluate the trainability and expressivity of the model, and prune the less promising models reduce the search space (Xing iet al. 2222)). Although they share a similar object to our work, they still rely on the concrete train- ing process evaluate the model performance. Instead, we regard the cilee--md--------maaaee information as a knowledge base to retrieve the suitable models without actual training.
AutoML with Large Language Models. Large Language Models (LLMs), such as GPT-4 (LLM
2023) and I.LaMA (LLMM) 2024), have recently garnered significant attention for their capabilities, positioning them as powerful tools for automating various tasks (including model designs). Un- like traditional AutoMI. methods that rely heavily on search-based techniques, LLMs offer a more flexible and scalable alternative by leveraging their vast pre-trained knowledge to suggest model
structures based on textual descriptions of the task at hand (Wei et al. 2023 Wang et al. 2023a
Dong et al. 2023 Wang et al. 2024).
In the context of AutoML, LLMs interpret human-specified requirements through natural lan- guage prompts, enabling them to generate corresponding neural architectures tailored to specific tasks (Wang al. (002)). For instance, users can guide LLMS to produce customized neural net- work configurations by providing detailed prompts that include dataset characteristics, desired per- formance metrics, and architectural preferences. Some approaches integrate LLMs as controllers within a NAS framework, where the LLM generates candidate architectures that are then evaluated and refined iteratively (Zheng et al., 2023 Zhang et al., 2023 Wang al. 2023ap. Specifically, by leveraging their few-shot learning capabilities (Brown et al. 2020), LLM controllers can uti- lize the optimization trajectory written in text to decide which model to validate next, incorporating
additional domain-specific knowledge as a reference (Zhang et al.] 20244 Nasir et al. 2024: Wang jet al. 2024) for example, recognizing that the GraphSAGE (Hamilton et al.) a 201 model bet-
ter suited for handling dense graphs. Additionally, LLMs can assist in generating code for model implementation, further streamlining the development process (Cheng et al. 2023)..
Despite these advancements, challenges remain in fully leveraging LLMs for AutoML tasks. LLMs may lack detailed domain lomain-specific knowledge required for tuning hyperparameters or may generate architectures that common but not necessarily optimal for given dataset (Wang et al. 2024). Integrating LLMs with structured knowledge sources, such as KGs or deep learning benchmarks, can enhance their capability produce more customized and effective neural network designs.
3

---

Published as a conference paper at ICLR 2025
Dataset
foz perfemmaaee
Canegonical Festures of Madel
thame
+Nome
+Posperty
-Prsperty
hot HPC
Hyper-parom (Optimizer)
Nanerical Festuncs
has, performance
Nurerica Features
-Semantic Descrigtion
Extity
-5Db/Adm
-Torget Enfltty
.1b
-IB .Property
Gategomica Dataset
Perr (Ace. Recell, Latercy?
(SPU/CPU/OS)
Domain
Nur Forems
has sa-dwore
+Nmee CS/Plysics
hoc_ Semon
<hame
KTXX000//410
Tosk
.Nime
Architzcture
Multi/sngee lobel
(Arch Graph/Loyer Openator)
Hoterogeneity
heteragone
+Ib
rossiiy
+Noee
.Desc
Modsliit
archituchure
<Nome
Homo/Hitee
fai_roabilly
.ID
-Nome
GAT/GGN
Figure 2: An architectural diagram of the Knowledge Benchmark Graph.
PROBLEM DEFINITION
In equation 1 it is quite time-consuming to obtain the accurate performance (M Dz)] } of many candidate models F E F on an unseen dataset D To efficiently obtain (F;;; D when searhc- ing for F², some classic methods have been proposed in the early years. For example, Domhan ett al. (2015) propose to explore the learning curves to predict the performance of model on dataset. Furthermore, one-shot NAS proposes the weight sharing mechanism to avoid the compu- tational overhead of training many candidate models (Pham et al., 2018). However, these methods still require a negliggie computational overhead, since they all need to train models somehow. Inspired by many well-established benchmarks Chitty- Venkat- et al. (2023), we may recommend those benchmark models Fb E F²' that are effective for benchmark data sets Do E ID' to unseen data D In such a way, the complex model design problem in equation Caan be simplified as follows.
Definition 1 (The Problem of Retrieving Models from Benchmark to Unseen Data) Given an unseen dataset D and a model search space Fo with performance records on benchmark datasets D', the goal is to find an effective model Fo for D by measuring the similarity between benchmark D with unseen Da, and retrieving most effective models from F.;
max MMX:M(F;;>) O max FEF
max
S(D", M(F: - D"),
(2)
FER
where S(D', D2) evaluates the similarity between benchmark data set D6 with unseen data D
There are two main intuitions behind Def. First, if the given data is one have seen before, only need to find the one model with the outstanding performance from a well-established bench- mark record (see Sec. 4 Secondly, as we only need to design models for unseen data D", we can find the benchmark dataset D E IDdh that is similar to D and then transfer the most effective mod- els from 1s on Dd D This approach aligns with the intuitive principle: "Similar datasets prefer similar models" [Bardenet et al. 2013). It worth noting that transfer learning (Pan & Yang 2009) focuses more on how reuse specific model in a source domain on target domain to alleviate the data scarce issue. Def. [ focuses more on how to quickly recommend a model F from massive benchmark models Fb and data sets D that can achieve high performance on unseen data D
To leverage benchmark data, two challenges need to be addressed: 1) Organizing Benchmark Data: The organization of benchmark data largely affects storage efficiency and information re- trieval effectiveness. We address this challenge by proposing the knowledge benchmark graph (KBG) in Sec. 4 2) Retrieve Effective Models: As the benchmark knowledge does not directly translate unseen datasets, developing robust S(-) that prioritizes the most relevant Db for D and selecting the most suitable models to D based on the identified similar datasets are non-trivial tasks. We propose the retrieval method in Sec. 5 address this challenge. Since S(-) is the founda- tion of this novel retrieval process, we further propose a comprehensive evaluation metric in Sec.
4 BENCHMARK KNOWLEDGE GRAPH CONSTRUCTION
Benchmark data, traditionally stored in tabular formats YYing et al. 2019a Dong & Yang 2020 Qin al. 2022), contains valuable information about model performance across various datasets.
4

---

Published as a conference paper at ICLR 2025
Using traditional tabular formats to represent diverse model structures across multiple datasets poses challenges, such as missing values and inefficient storage, which complicates data management and retrieval. To better utilize the benchmark data, transform it into Knowledge Benchmark Graph (see Fig. 2). The KBG can organizes datasets, models, and performance as entities and relations, enabling flexible, context-aware queries based on relationships between entities rather than simple row-column searches. This structure improves data accessibility and interpretability. allowing LLMs process and respond to queries more effectively. We next introduce the KBG in detail.
Entity and Its Attributes. To record the "dat-mmodel-eerfommace""" information, the KBG mainly has following entities:
1. Data: As fundamental component, Data: fdata_ID, data Name, dattaPropeeeeee
records information about the datasets.
description:(-.. are the attributes that provide high-level information about the dataset, including statistical features and semantic description. Detailed feature information and examples are provided in the Appx|A.3
Model: A neural network model generally consists of three main parts: model architec- ture, hyperparameters, and software/hardware configurations. To reduce redundancy, KBG uses Model: : {model.ID, model_Name, model ropertyy to store models and model- specific attributes (typically numerical, such dropout ratio), while common attributes are stored as other entity types (see Other Entity Types for more details).
model P Property: (numer cal Features: (...)) } are the attributes that include the numerical configurations of the model, such as learning rates, epochs, and hidden units.
3. Other Entity Types: These entities store categorical features of datasets and models, using the format ent itt Type: (ent TD, entity Name) with actual entity names introduced below. Generally, they can be divided into three groups.
Common Attributes: task and modality
essential for determining the relatedness of datasets and models and for selecting suitable ones.
Data-related Attributes: Semantic aspects the datasets, such as their domain and hete heterogeneity (for graph datasets), represented entities.
Model-related Attributes: Important categorical features for models include software/hardware information like operat ng_Syssem, CPU, and GPU, model architecture components like architecture.Graph and Layer.Operator, and hyperparameter configurations like optimizer
Because the categorical features are widely shared across datasets and/or models, transforming them into entities reduces the complexity in the storage and allows higher extendibility for cate- gories. On the other hands, the numerical features and semantic descriptions are often unique to the corresponding entities, thus keeping them in the attributes is more applicable.
Relations. There are two types of relations in the KBG:
Performance: KBG utilize the hyper-relational facts (Rosso et al., 2020 r(en, ets {r"
different models on various datasets, such 1)))), to store the performance wtw.ooddee....cccaacca.aaaaaaaaa (Accuracy:.... Loss pooss:...,. ................................ } Gen- hasPerformance (data_ID, model_ID. erally, hasPerformance (data.ID, mode I_ID) is regarded as main triplet, while (Accuracy: Loss: are role-value pairs {ro oo)N
Feature: The relations between the datasets/models and their kinds of properties are stored as bi- nary facts hasFeature: (data ID/model.ne ID, feature ID), where the actual relation names vary with the corresponding features.
As shown in Fig. 2 such relations include has_domain, has.task, has_modality. has hasaarchieecture has hardwaree and has SPPO etc.
RETRIEVING MODELS WITH KBG
Based on the KBG constructed in Sec. 4 it is now important to retrieve the similar datasets and and relevant models for unseen data. Thus, we design dataset similarity score S(,,) and model relevance score R(-,)) for retrieval correspondingly.
Data Similarity. Following Def. designing an effective dataset similarity score S(.) is the crucial part. As first approach, directly use the statistical features of datasets measure the similarity.
5

---

Published as a conference paper at ICLR 2025
Let the feature vectors of the benchmark dataset Dd and the unseen dataset D" be represented as d-dimensional vectors v and v", respecttively. The dataset similarity score S(vv, v") is defined as:
v(v,,"))==1 d Σ min -vy d'sd
where represents the maximum allowable difference for any feature dimension, and is power parameter that controls the sensitivity to feature differences. Both a and p are hyperparameters. The similarity is ranged within [o, with a higher value for more similar dataset. Empirically, we set and a = 0.8 to mitigate the impact of outlier feature differences.
Model Relevance. While -(-.-) helps prune the model search space, directly transforming models from the most similar D to D can not guarantee a good performance. Moreover, a less similar dataset may also assist defining the model performance on D Thus, to better decide the relevance of model F to D it is necessary to aggregate the model performances multiple benchmark datasets that are similar to D The model relevance score R(...) defined as:
R(F;D")= = E M(F:D')- SDD",
(3)
D'ED*
where D {D*IS(D*, D*) 2 8, Dd D²). R(-,-) is a reflection of Def. which captures both the model's historical performance and the similarity of the benchmark dataset to the unseen dataset.
Please note that this paper focuses more on how to avoid training as much as possible by leveraging the prior information of KBG itself. So that we only propose relatively simple S(D*, D") and (F;; D to demonstrate the feasibility of the proposed method. The more advanced and accurate designs of the scores can be easily combined with ours with a computational overhead.
EVALUATING DATA SIMILARITY
As discussed in Sec. the quality of data similarity metric S(D*, D") largely affects the model retrieval performance. However, given the novelty of the problem, there lacks a comprehensive evaluation metric to assess the effectiveness of S(D*, D"). While the performance on D is an in- tuitive measurement, it does not directly indicate whether the model transfer is successful. Thus, we first introduce factors that directly reflects the transfer performance retrieved models in Sec. 6. Then based on these factors, propose a novel evaluation metric for S(D*, D") in Sec. 6.2
6.1 FACTORS OF DATA SIMILARITY EFFECTIVENESS
When retrieving models from the KBG for D sufficiently similar Dd datasets should yield models that perform well on D Thus, the effectiveness of S(D', D²) is measured by how well top- performing models from D transfer to D Formally, the problem can be defined as follows:
Definition (The Problem Transferring Models in KBG Unseen Data) Given an unseen dataset D and a model search space IF' with performance records on benchmark datasets D stored in KBG, the goal is to design a dataset similarity metric S(.) such that, if S(D', Dz) SDD 2 8 and subset of models satisfies IF = {F E Fb MMFF; Dd) 2 Th then the probability that M(F; D is close to M(F;; D*) (within an error tolerance e) is at least A:
PMM(F:D") 2 ((FF
(4)
where 8 and 7 are the thresholds for similarity and performance, is the error tolerance, and A is the probability lower-bound.
Following Def. 2 S(-,-) is evaluated by two key factors, e and A. The value e measures how well transferred models maintain high performance, while A represents the probability of successful transfer given e Intuitively, smaller error tolerance lowers the success probability A and an effective -(.,)) should minimize while maximizing A.

---

Published as a conference paper at ICLR 2025
curve different
We further study the relationships between and A with .(:..) defined in equation and the KBG constructed from NAS-Bench-Graphe (Qin al. 2022). We select one dataset from KBG as 1 and treat the most simi- lar remaining dataset as Do_ In order to compare per- formances across different datasets, we define M(-) as relative measure of performance, comparing model J to the theoretically optimal model F in F", M(F; D*): (aD"(F
citeseer
pubmed
physics
(Fn:(F";Dby We transfer models with performance no
computers
arsiv
less than 0.990 on D to D and compute the trans- ferred performance error and confidence A. As shown Fig. 3 A increases monotonically with e, and when is within 0.04, A is at least 20% for all datasets. This also confirms the feasibility of transferring high- high-performing models to similar datasets.
0.08
Figure 3: The probability that the trans- ferred model's performance is within
Remark 6.1 How to design a good similarity metric S(D Dn) is an open question. The core idea of Def 2 is that better similarity metric should lead to smaller but larger A equation | The Deft serves as an guidance for subsequent work to evaluate different S(D", D").
6.2 DATA SIMILARITY EVALUATION
Following Def. 2 retrieved models Ie from D to D" have a better transferability if they have higher possibility A of being transferred with a low performance drop e. This equivalent to lifting the 6-A curve to the upper left corner and results in large area under the curve, illustrated in Fig. Thus, we can quantify the transferability T-) of F by measuring the area under the e-A ( fmar is hyper-paramene
that limits the largest transferred error, too large error is meaningless. Considering that the exact relationship between and A is unknown and needs to be empirically evaluated, we use the estimated version of T--) in practice:
curve, i.e. T(F9) == Jo mea
Emaz
T(F)
Σ A(e),
(5)
where N is the number of estimated values. more transferable Ie has a larger score close to 1, showing the performances of F sustain across two datasets. This score serves as the ground truth similarity between D and D and good S(Db, D should align with this observation.
Then, given a set of benchmark datasets } and set of candidate models {Ff): separately selected from each benchmark dataset with the same standard, we evaluate S(.:-) by the linearity between the similarity and the transferability. Following the definition of R score, define the Relevance Linearity Score (RLS) as:
RLS(S) 1 (pEED;;)(((D) D*) ((F)))²
EDJE(D;) (S(D), D*) - Fovg)²
EnpEnne (T(F)) is the average transferability score of (ET).. A higher RIS indicates a better linearity between the similarity and the transferability, suggesting that -(.-) correctly ranks the dataset similarity by the model performances the unseen datasets.
where Fang 1/
EXPERIMENT
7.1 EXPERIMENT SETUP
Implementation Details. To verify the effectiveness of KBG and integration LLMs with KBG, the benchmark datasets used in our experiments is the NAS- Bench-Graph dataset (Qin al. 2022).
Table 1: The statistics of KBG
Models # Datasets # Perf. Rec. # Entities] # Facts
26,206
211,669
211,712
2,540,064
7

---

Published as a conference paper at ICLR 2025
Table 2:
The initial performance comparison (absolute accuracy) of GNN designed by different works. We mark the best performance in bold, and and second best in underline.
Type
Model
Cora
Citeseer
PubMed
CS
Phys.
Photo
Comp.
arXiv
GCN
80.97
69 69.90
77.46
88.65
90.85
89.44
83.16
71.08
GAT
80.83
70.70
75.93
88.72
89.47
89.93
81.35
71.24
SAGE
79.47
66.13
75.50
87.81
91.43
88.29
81.46
70.78
Manual
GIN
79.77
63.30
76.74
81.08
86.67
87.37
73.95
61.33
ChebNet
79.40
67.03
75.13
89.50
89.75
86.65
79.10
70.87
ARMA
78.33
66.20
75.00
89.87
88.88
86.55
78.47
70.87
k-GNN
78.06
30.97
75.38
83.81
88.98
86.45
76.31
63.18
Random
77.87
66.64
74.16
81.78
90.59
89.04
76.61
68.93
Classic
78,23
66.40
72.88
87.03
88.07
87.30
77.56
68.28
RL
73.44
65.35
75.44
86.17
88.15
89.48
77.70
68.00
GNAS
78.55
63.25
73.04
86.04
89.54
87.27
70.96
69.94
Auto-GNN
78.58
65.60
76.07
89.06
89.26
89.34
77.49
70.62
AutoML
GPT4GNAS
78.50
67.46
73.89
89.26
89.44
89.12
77.21
68.98
GHGNAS
79.13
67.35
74.90
89.15
88.94
89.42
77.04
69.66
DesiGNN-ini
80.31
69.20
76.60
89.64
92.10
91.19
82.20
71.50
Kendall
67.73
69.20
71.80
88.56
91.56
88.90
76.85
71.49
Sim.
Overlap
79.36
67.30
71.80
88.
89.95
90.37
76.85
71.68
Ours
82.53
69.20
76.53
89.32
90.34
90.37
76.61
71.68
evaluated across 9 differ- ent graph datasets. The search space consists of GNN model architectures, represented as directed acyclic graphs (DAGs) with nodes and 9 possible layer types. The 9 datasets used in the evalua- tion are: Cora, Citeseer, Pubmed, CS, Physics, Photo, Computers, Arxiv, and Proteins. The detailed statistics are provided Tab and Appx. A.3
which includes performance records for 26,206 GNNs (Zhou et al. 2020
Evaluation Metric. The performance of suggested model design F E F on an unseen dataset D evaluated as its accuracy M(F; D"). For retrieval methods, F is directly retrieved from KBG based on the performance records of similar datasets D DV, i.e., removing the records to related to D from KBG. While for AutoMI. methods, generated by the their algorithms.
Baselines. We compare our approach against diverse range of baselines:
Manually Designed GNNs: We include several popular GNNs commonly used in the literature, such as GCN (Kipf & Welling 2017), GAT (Velickovic et al.) 2017). GraphSAGE (Hamilton et al.
2017b), GIN (Xu et al.] 2019) ChebNet (Defferrard et al. 2016), ARMA (Bianchi et 2019
and k-GNN (Morris et al. 2019).
Classical NAS Algorithms: We compare against Random Search Li & Talwalkar 2019). Evo-
lutionary Algorithms (Real et all 2019), and Reinforcement Learning (Zoph & Le 2017).
AutoML Methods for GNNs: We include state-of-the-arr AutoML methods specifically designed
for GNN architecture search, such as GNAS and Auto-GNN (Gao et al. 2020 Zhou et al. 2022
LLM--ased AutoML Methods: Given the recent success of large language models (LLMs) in AutoML, we include GPT4GNAS (Wang et al.) (023a), GHGNAS (Dong et al. 2023 and DesiGNN (Wang et al. 2024 as L.LM-based AutoMI. baselines.
Similarity Metrics: As retrieve model designs based data similarity in Sec. against other similarity metrics such as Kendall Youu et al. 2020) and Overlap (Wang et al., 2024),
compare
which measure similarity based on performance differences of one model across different datasets.
7.2 EFFECTIVENESS OF RETRIEVING MODEL DESIGNS WITH KBG
The key to applying the KBG is whether an effective model from a similar dataset is still effective on the unseen dataset. To validate this, we simply retrive the best performing model from the most similar benchmark datasets to the unseen data, and report its performance on the unseen data. As shown in Tab. our naive retrieval method achieves the best or second-best performance on 4 out of 8 datasets, and is competitive on the remaining 4 datasets. Given that other methods require repeated training, which time-intensive, this result demonstrates the effectiveness of using KBG to as- sist model design. Additionally, compared to Kendall and Overlap, which rely supervised model
The code is available
uuhnnoo321 32/ /
8

---

Published as a conference paper at ICLR 2025
Table 3:
The RLS of different similarity metries in initial model suggestion.
Metric
Cora
Citeseer
PubMed
CS
Phys.
Photo
Comp.
Arxiv
L1
0.452
0.126
0.003
0.020
0.013
0.310
0.025
0.014
1.2
0.498
0.064
0.012
0.007
0.040
0.328
0.036
0.012
Ours
0.435
0.340
0.004
0.025
0.002
0.252
0.043
0.027
cora
physics
aniiv
8.8
0.8-
0.0
0.6
0.6-
0.6
04
oteseer 0.5303
0.4
0.4
physics
photo-0.5280
and 6170
c5-0.5772
02
compitiessn: 5072
0.2
pubmeed--... 5858
0.2
pubmed--0 5649
pubmed 0.4935
photo 0.5492
photo 0.5065
3.0
0.02
0.04
0.05
0.05
0.10
0.02
0.04
0.06
0.08
0.19
0.02
0.04
0.06
0.08
0.10
Figure 5: 6-A curve when considering Cora, Physics, and Arxiv unseen datasets. The legends denote the benchmark datasets and their similarites to the unseen datasets. T is fixed to 0.980.
performance on unseen data, our metric achieves better performance with only intrinsic dataset char- acteristics. This highlights the effectiveness of leveraging dataset characteristies for model transfer.
7.3 THE STUDY ON THE IMPACT OF T, 6, AND A
With the dataset similarity metric validated as effec- tive for model design, we further investigate the def-
andv
anxiv
1.0
T-0.983
0.9
T-0.980
Rerferaacee
0.734
T- 986
T-0.980
0.8
T.0.983
inition of model transfer- ability based on Def[] Re- call that effective can- didate model set IFe (2 7) should have a high suc- cess probability (A) with a
0.7
t=0.58
T-0.955
T-0.669
0.8
--0.992
0.730
0.5
0.08
0.10
(b) Effecc r on the performance. Figure 4: Effect of performance threshold J on e, A and performance.
(a) Effect T on e and
low error threshold (e). As shown in Fig. 4 we study the impact of increasing T on and A. Fig. 4a llusttrtes that maintaining a high A at a low c is challenging, trend consistent across all values of т. However, when com- paring Fig. 4a to Fig. 4bb we observe that the transferred models with smaller generally exhibit better performances. This underscores the importance pursuing a small for successful model trans- ferability. Additionally, Fig. [bb supports assumption that better setter-performing model (larger r) from a benchmark dataset is more likely to perform well on an unseen dataset. The figures on other datasets presented in the Appx
7.4 THE STUDY ON THE SIMILARITY SESIGN WITH EVALUATION METRIC
As shown in Fig. we study how the similarity between D" and D' affects the model transfer performance. Intuitively, a more similar Dd should have a higher possibility A transfering models within a low performance drop 6, which is reflected in a larger area under the e-A curve. The results in Fig. 5 show that the current similarity metric provides a generally accurate ranking for the top three similar datasets. However, still fails rank the truly most similar dataset as the highest.
Following RLS defined in Sec. 6.2 we next present scatter plots between transferability score 7 7(F") ie. the E-A curve area, and dataset similarity in Fig. 6 We expect the curve area to be linear to the similarity, but the results indicate that the current similarity metric flawed in achieving this goal. However, when comparing the RLS (R² score) with the model performance in Tab. 2 we observe that when the metric has a higher RLS on an unseen dataset, the retrieved similar dataset tends to have more transferrable model design. For example, our S(-,-) has higher RLS Cora, and the retrieved model has the best performance as well. This validates the effectiveness of using linearity as an evaluation metrie for dataset similarity. Furthermore, we evaluate the different designs of -(..)) with RLS in Tab. 3 It can be seen that no single metric is effective for all datasets. More detailed comparisons and analysis are provided in Appx. A.7

---

Published as a conference paper at ICLR 2025
cora o.pp- R3w0.430
physics Slape- 0,049, R'=0 001
anxy Slape -0214, R'u0.027
0.5
getmmad
gomputery
computers
loseer
07
ghote
glesse
08
65% Sysics
gemputers
0.75
ghisto
giones
60ra
0.65
000
0.46
348
0.50
0.52
0.50
0.55
0.00
0.65
0.45
0.50
0.55
0.60
dotaset similarity
cbtsset Simlarkt
dacaset similarty
Figure 6: The linearity between the dataset similarity and the area below c-A curve. The R score
denotes the RLS metric.
7.5 IMPACT OF USING DIFFERENT SIZES OF Dh ON MODEL SUGGESTION
The previous experiments focused recommending models using single similar dataset. How- ever, the KBG structure allows recommendations from multiple similar datasets. In Tab. 4 we analyze how incorporating different datasets affects model performance with R(-) defined in equa- tion 3 When more similar datasets are considered, the recommended models can outperform those from initially less optimal datasets. However, the improvement is inconsistent, likely due the cur- rent simplistic combination method. With more sophisticated approach to measure data similarity, leveraging multiple similar datasets is expected to result better model recommendations.
Table 4: The effectiveness of model relevance score R(.) with different [D] |D|=1
=1 is the same as the naive recommendation. We mark the best performance bold, and and second best underline.
Method
Cora
Citeseer
PubMed
CS
Phys.
Photo
Comp.
arXiv
DesiGNN-init
80.31
69.20
76.60
89.64
92.10
91.19
82.20
71.50
R(-).
=F==1
82.53
69.20
76.53
89.32
90.34
90.37
76.61
71.68
R
-2
79.16
68.40
75.63
89.57
92,05
91.19
83,39
70.33
(
R
DV
=3
79.63
68.73
76.46
89.45
92.11
91.48
81.02
71.09
7.6
INCOOPERATION WITH LARGE LANGUAGE MODEL
As an important application scenario KBG, we present the study on the effectiveness of incoop- erating LLMs with KBG suggest models in Tab. 5
Given unseen dataset, we first retrieve candidate models Fe from KBG with R(-) in equation 3 Then we require the LLM to infer or select model F" from F.. Afterwards, we repeatedly retrieve another set of candidates whose architec- tures and performances across ID' are both similar to F" and ask LLM to refine F'. Results show that combining KBG with LLM outperforms the initial models from DesiGNN and only KBG. Be- sides, it can be seen that selecting a model design with LLMs is more effective than inferring one. This shows that retrieved models are already effective, and forcing L.LMs infer a new model design may not be necessary. Experiment details are provided in Appx
Table 5: The performance of incooperating LLMs with KBG to suggest models.
Type
Method
Cora
Citeseer
PubMed
CS
Phys.
Photo
Comp.
arXiv
DesiGNN-Init
80.31
69.20
76.60
89.64
92.10
91.19
82,20
71,50
Initial
KBG-Init
82.53
69.20
76.53
89.32
90.34
90.37
76.61
71.68
KBG+LLM Infer
80.32
69.56
76.60
89.53
92.61
91.87
81.26
71.69
Refined
KBG+LLMM Select
80.54
69.06
77.23
89.53
92.61
91.95
82.34
72.05
CONCLUSION
In conclusion, the Knowledge Benchmark Graph (KBG) presents novel approach to enhance neural network design by structuring benchmark data into a graph that stores model performance across various datasets. The KBG facilitates efficient and automated model designs for unseen datasets by retrieving insightful knowledge from prior benchmarks. However, this paper still leave several challenges unsolved. For example, we may construct more accurate similarity functions between seen dataset and unseen one. Besides, it is meaningful to combine existing benchmarks together to build larger KBG to support model design different domains. the subsequent research, we will aim combine KBG with automatic learning systems and take step towards smarter and more efficient neural network design process.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Lei Chen's work is partially supported by National Key Research and Development Program of China Grant No. 2023YFF0725100, National Science Foundation of China (NSFC) under Grant No. U22B2060, Guangdong-Hong Kong Technology Innovation Joint Funding Scheme Project No. 2024A0505040012, the Hong Kong RGC GRF Project 16213620, RIF Project R6020-19, AOE Project AoE/E-603/18, Theme- based project TRS T41-603/20R, CRF Project C2004-21G, Guang- dong Province Science and Technology Plan Project 2023A0505030011, Guangzhou municipal- ity big data intelligence key lab, 2023A03J0012, Hong Kong ITC ITF grants MHX/078/211 and PRP/004/22FX, Zhujiang scholar program 2021JC02X1 Microsoft Research Asia Collaborative Research Grant, HKUST- joint research lab and 2023 HKUST Shenzhen-Hong Kong Col- laborative Innovation Institute Green Sustainability Special Fund, from Shui On Xintiandi and the InnoSpace GBA.
REFERENCES
Gpt-4 technical report. 2023. URL 257532815
The llama herd of models. ArXiv, abs/2407.21783, 2024.
URL. htttps://ap
semant. i csscholae ar. corp
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michèle Sebag. Collaborative hyperparame-
ter tuming. In International Conference on Machine Learning, 2013. URL https: //api.
semant wescholar........oo org :ttpg//ooosD:::::0 8791227
Filippo Maria Bianchi, Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. Graph neural networks with convolutional ARMA filters. CoRR, abs/1901.0133 001343,, 2019. URL httl:/tttapxii....t// org/
1b0/1901. 01343
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpps:://ppcccceigg....tt 115570688.00441711888888.2222222
neurips html
Liying Cheng, Xingxuan Li, and Lidong Bing. Is GPT-4 good data analyst? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics; EMNLP 2023, Singapore, December 6-10, 2023, pp. 9496-9514. Association for Computational Linguistics, 2023. doi: 10. 10.118653/11/2223311. FINDINGS FINGSIII FINDINGG--EELLP66.1 EMNLP6337. URL htttp:://doo
org/10. 10/ 100.186533.0 2010665531122220 findings www.aniinnnneenp...mmmmmm emnlp. 637
Krishna Teja Chitt- Murali Emani, Venkatram Vishwanath, and Arun K. Somani. Neural architecture search benchmarks: Insights and survey, IEEE Access, 11:25217-25236, 2023. doi: 10. 10.110//ACCESS.222.3355588
URL
2023.
3253818
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural net- works on graphs with fast localized spectral filtering. Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Infor- mation Processing Systems 2016, December 5-10. 2016, Barcelona, Spain. pp. 3837- 3845, 2016. URL http:tts/pppccccinns..eeeeee httpp//wwwp.....p.a////////////. 0
Daniel D. Lee, Masashi

---

Published as a conference paper at ICLR 2025
Shimin Di and Lei Chen. Message function search for knowledge graph embedding. In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 2633-2644, New York. NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394161. doi: 10. 10.1145/354300735883566
URL
114. 354555554557777 3583546
Shimin Di, Quanming Yao, and Lei Chen. Searching to sparsify tensor decomposition for n-ary
relational data. In Proceedings of the Web Conferencee 2021, wWW '21, pp. 4043-4054, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 7781455333127 doi: 10. 1145/3442381 3449853. URL https::/ http:////t...g9/01.111110.
.rg. 101100..115/200 3448144 ..4.442211111.48...1 33449833
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparam- eter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence, 2015.
Haoyuan Dong, Yang Gao, Haishuai Wang. Hong Yang, and Peng Zhang. Heterogeneous graph neural architecture search with GPT-4, CoRR, abs/2312.08668/ 2023. doi: 10.48550/ARXIV. 2312.08680. httptt::p//iiiioo/ooo.oooo/// 3tth:/00.485555/38 2312 08680
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020. URL https:
//openzeview. wttt/orrmmia--uuee......
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-Bench: Benchmarking nas algorithms for architecture topology and size. IEEE Transactions on Putlern Analysis and Ma- chine Intellignnce (TPAMI). 2021. doi: 10009/TPAAM.2221335455553 doi 10 1109/TPAMM 2021 3054824
Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu, Graph neural architecture search. In Christian Bessiere (ed.), Proceedings of the Twenty-innh International Joint Conference on Artificial Intelligence, IJCAI 2020. pp. 1403 1409. ijeai.org. 2020. doi: 10.24963/ICAA.22200 195. https:/ //ooi org/ 10 24 9943 i jcat 2020/195
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. In Forty-first International Conference Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: http:::///ppeeevviiiie ew net. forumrrrmmmmmm ?
LfJgeBNCEF
William L. Hamilton, Zhitao Ying, and Jure Leskovee. Inductive representation learning on large graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference Neural Informa- tion Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1024 1034, 2017a. URL https: http:::::/ppocceedieo neuriis............. httpp/pppr//1//haaapa wew.aa/a/weeeeeaaaaaaaaaa www.eebbaabbssasa.tabeeeaaaaee.e html
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neural Informa- tion Processing Systems 2017, December 4-9, 2017, Long Beuch, CA, USA, pp. 1024 1034, 2017b. URL https: http::://ppoceeddnngg hew.iis..e.ippppp wwtparissss.c...eeee wttpppppp///////aa//ssesee Badddaaecaaaddbbaaaaaaaaaaeee Abstrrattt-bttttttaattaa .tml
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art, Knowl, Based URL
Syst., 212:26 2021. doi: 10.1016/J
202.:............ 106622
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual
12

---

Published as a conference paper at ICLR 2025
Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL htRpiptpps//ppocccaans...sos html 1aoodd11acchh77222233222222666
httpsecccpppppee cc/papprr/222/haaa.
Udayan Khurana, Horst Samulowitz, and Deepak S. Turaga. Feature engineering for predictive modeling using reinforcement learning. ArXiv, (0s/0709007,50, 2017. URL https://api.
semant cscho ar. ort.CorrpssI: 8175444
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htLhttps: //openzeview.oo net/ wot.e//ooom..aaSSAaaaaaaa.
Liam Li and Ameet Talwalkar. Random search and reprodueibility for neural architecture search. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pp. 367-377. AUAT Press, 2019. URL http:::/ppoceeedinge mlm.mmm......
html.
Shiqing Liu, Haoyu Zhang, and Yaochu Jin. A survey computationally efficient neural architec- ture search, 2022. httt::
org/ abs/ absaaa////22 2206. 01520
Vasco Lopes, Saeid Alirezazadeh, and Luis A. Alexandre. EPE-NAS: efficient performance es- timation without training for neural architecture search. In Igor Farkas, Paolo Masulli, Se- bastian Otte, and Stefan Wermter (eds.), Artificial Neural Networks and Machine Learning ICANN 2021- 30th International Conference on Artificial Neural Networks, Bratislava, Slo- vakia, September 14-17, 2021, Proceedings, Part V, volume 12895 of Lecture Notes in Com- puter Science, pp. 552-563, Springer, 2021. doi: 10. 100000/9778---000000 00188-8--000--88--88-88-000 htttss://di.... 011.:/00000000110000000003333333
URL
44
Joseph Charles Mellor, Jack Turner, Amos J. Storkey, and Elliot Crowley. Neural architecture search without training. CoRR, abs/2006.046477, 2020. URL
wrg. abs/
2006 04647
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In The Thirty-Thine AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Inno- vative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, Jan- uary 27 February 1, 2019, pp. 4602 4609. AAAI Press, 2019. doi: (0.1609/AAALL330 V33101. 33014602. URL. https:::: / /o.. 100 609/aaai v33101
33014602 .33111133016162022
Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias Boutros Khalil, and Deepak S. Turaga. Learning feature engineering for classification. In International Joint Conference on Artificial Intelligence, 2017. URL matpps://aa //api. semanticschene
wwt. Corg/CorpsII::
19165886
Muhammad Umair Nasir, Sam Earle, Julian Togelius, Steven James, and Christopher W. Cleghorn. Llmatic: Neural architecture search via large language models and quality diversity optimization. In Xiaodong Li and Julia Handl (eds.), Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2024, Melbourne, VIC, Australia, July 14-18, 2024. ACM, 2024. doi: 10. (145/3638529.365440
URL/attps:////
wttpl//../.......// /tt.///33335522555511111 3654017
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IFFE Transactions on knowledge and data engineering, 22(10):1345-1333 2009.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. 4095-4104, PMLR, 2018.
Yijian Qin, Ziwei Zhang, Xin Wang. Zeyang Zhang, and Wenwu Zhu. Nenche graph: Benchmarking graph neural architecture search. In Sanmi Koyejo, S. Mohamed,
13

---

Published as a conference paper at ICLR 2025
A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Infor- mation Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022. NeurlPS 2022, New Orleans, LA. USA, November 28 December 9, wip.e 2022, 2022. URL http://ppper..tssscccppppppppeep
fash/004bed4h oaw.a//aaraaaaddee1c2222-----aa
Ben.hmarks..he
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019. The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaiï, USA, January 27 -February 1, 2019, pp. 4780 4789. AAAI Press, 2019. doi: 10. 10.1609/AAAI. V33101...3331333 1330141033
URL https: 0tttp:://dol.org11..... 1tt.:....16.9. aaai...a............aa.....aa... v33101.
33014780
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Yao (Bernie) Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. comprehensive survey of neural architecture search. ACM Computing Surveys
(CSUR), 54:1 34, 2020. URL ot
219305190
Paolo Rosso, Dingqi Yang, and Philippe Cudré-Mauroux, Beyond triplets: hyper-relational knowl- edge graph embedding for link prediction. In Proceedings of the web conference 2020, pp. 1885- 1896, 2020.
Imrus Salchin, Md.Shamiul Islam, Pritom Saha, S. M. Noman, Azra Tuni, Md.Mehedi Hasan, and Md Abu Baten. Automl: A systematic review automated machine learning with neu- ral architecture search. Journal of Information and Intelligence 2023. URL /htpps://ppi.
semant www.olccchhcccoooooo ar. org. wotg/Corpuustiiiooooooooor ID: 16:433334444
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi-Rad Collective classification in network data. A/ Mag., 29(3)933- 2008 doi: 10. 10.1609/AIMAG.
V2913.2157. URI.https:: /etttts::/doi.ooo http:///ioooog///////// 100 0. 609/ wamaa...a... v2913.213291..1 .2577
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2016. doi: 1 109/JP/C..2015.224....
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. CoRR, abs/1811 1.05668,, 2018. URL htttp://arx: org/abs/ 1811111111111 05868
Zhi-Hao Tan, Jian-Dong Liu, Xiao-Dong Bi, Peng Tan, Qin-Cheng Zheng, Hai-Tian Liu, Yi Xie, Xiao- Kiao-Chuan Zou, Yang Yu, and Zhi-Hua Zhou. Beimingwu: A learnware dock system. In Knowl- edge Discovery and Data Mining, 2024. URL pattps://api. wwwanticscoool............. orr/
CorpusID:267:0
Hidenori Tanaka, Daniel Kunin, Daniel I.. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. ArXiv, abs/2006.05467, 2020. URL
bittps://apt www.nt ottpa//eee...aa.....oooooooooorr 219588821
Alexander Tornede, Difan Deng, Theresa Fimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne Theoddrakoppollss, Tanja Tornede, Henning Wachsmuth, and Marius Lin- dauer. Automl in the age of large language models: Current challenges, future opportunities and risks. Trans. Mach. Learn. Res., 2024, 2024. URL httpp:://peenreieew.cc 1 id=cAthubstyc
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liō, and Yoshua Bengio. Graph attention networks. CoRR, abs/1710.10003. 2017. URL/attp::/azziv..rrr
abs//11 /1se/17000 1..00.1999 10903
Haishuai Wang, Yang Gao, Xin Zheng, Peng Zhang, Hongyang Chen, and Jiajun Bu. Graph neural architecture search with GPT-4. CoRR. abs/2310.01436, 2023a. doi: 10.48550/ARXIV23100. 01436. JRL.ttts:: //do: laroiioogg/000 48550/ 2310
14

---

Published as a conference paper at ICLR 2025
Jialiang Wang. Shimin Di, Hanmo Liu, Zhili Wang. Jiachuan Wang, Lei Chen, and Xiaofang Zhou. Computation ffrinndly graph neural network design by accumulating knowledge on large language models, 2024. URL https: arxiv. www.aatwwii.gaaaaaaaaaaa wwwpp/////baaa/.eee../e/eaaeeeea abs/aaas//22 2408
Zhili WANG, Shimin Di, and Lei CHEN. Autogel: an automated graph neural network with ex- plicit link information. In Proceedings of the 35th International Conference Neural Informa- fion Processing Systems, NIPS '21, Red Hook, NY, USA, 2021, Curran Associates Inc. ISBN 9781713845393.
Zhili Wang, Shimin Di, and Lei Chen. A message passing neural network space for better captur- ing data-dependent receptive fields. In Proceedings of the 29th ACM SIGKDD Conference on
Knowledge Discovery and Data Mining, KDD '23, pp. Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/3580305.359999 35599244.
New York, NY, USA, 2023b.
htLphhtss://oo /05.:////...11153333330533..335 3599243
Zhili WANG, Shimin DI, Lei CHEN, and Xiaofang ZHOU. Search to fine-tune pre-trained graph neural networks for graph-level tasks. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pp. 2805-2819 19, 2024.
doi: 10. 109/CCDD000 (999/CCDE01166202444 46.2092.00219.
Lanning Wei, Zhiqiang He, Huan Zhao, and Quanming Yao. Unleashing the power of graph learning through llm-based autonomous agents. CoRR, abs/2309.04565. 2023. doi: 10.48550/ARXIV. 2309.04565. URL https::///datt http://tto.org//.//////o 48550/ 2309. 04565
Wei Wen, Hanxiao Liu, Hai Helen Li, Yiran Chen, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, 2019. https:/tp:://aat
semant www.chlll...oggoooo..ooo cccchholla..oooo/oorrrssst:::S2
Xi-Zhu Wu, Wen qi Xu, Song Liu, and Zhi-Hua Zhou. Model reuse with reduced kernel mean embedding specification. IEEE Transactions on Knowledge and Data Engineering, 35:699-710, 2020. UtLptttss: //api. semant icscholar: ottp/Corpss
Naili Xing, Shaofeng Cai, Gang Chen, Zhaojing Luo, Bengchin Ooi, and Jian Pei. Database na- tive model selection: Harnessing deep neural networks database systems. Proc. VLDB En- dow. 17:1020-1033, 2024. URL https: //api. semant cscholaa arr .ww. Arrg/CrrpusID. 268724517
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019, New Or- leans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttps://openrrviee..nro net/
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- 101ch-101: Towards reproducible neural architecture search. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Reseurch, pp. 7105-7114, Long Beach, Cali- fornia, USA, 09-15 Jun 2019a. PMLR. URL http: htttp::/proceeee http:://oooceeddeieoo ml r.press/v977 yingl9a.. html
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. In Kamalika Chaudhuri and Rus- lan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learn- ing, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7105 7105-7114. PMLR, 2019b. URL htR.httpp:/prrceeddngg mlr. wwt.ppess/v///ee http/////////n// html
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. In Hugo Larochelle, Mare' Aurelio Ranzato. Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. htt.//////ttss/ppoeeee....sooe. htt.://ennni.....o/////////////a fe
7oc.1e77ccceee. Abs
15

---

Published as a conference paper at ICLR 2025
Caiyang Yu, Xianggen Liu, Wentao Feng, Chenwei Tang, and Jiancheng Lv. GPT-NAS: evolu- tionary neural architecture search with the generative pre-trained model. CoRR, abs/2305.05351, 2023. doi: 10.48500ARRXV..235500555 2305 05351
URL https: ottts:://aol.org/1....388 w0tp///.....5//22111.....3
Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. MIcopilot: Unleashing the power of large language models in solving machine learning tasks. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, EACI. 2024 Volume I: Long Pupers, Julian' ss Malta, March 17-22, 2024, pp. 2959. Association for Computational Linguistics, 2024. URL
attps://aclanthology http://ww..oooooo.o//////aaaaaaa
.19]
Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou. Automl-gpt: Automatic machine learning with GPT. CORR, abs/2305.02499, 2023. doi: 10.48550/ARXIV. 2305.02499. U.L.
02499.
Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie. Can 10970,
GPT-4 perform neural architecture search? CoRR, abs/2304. 10970, 2023. doi: 10.4 2304.10970. URL https: /ww.i wrg/ 10. 48550 230..10997.77
Xiawu Zheng, Rongrong Ji, Qiang Wang. Qixiang Ye, Zhenguo Li, Yonghong Tian, and Qi Tian. Rethinking performance estimation in neural architecture search. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. (1353-11366 2020. URL https:
//api.semaaee cacchlar htwpeshaa.....rr//ooorroaraaTT/.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li. and Maosong Sun. Graph neural networks: review of methods and ap- doi: 10.1016/J A10PE.....0....21....0.0000 URL https:
plications. Al Open, 1:57-81, 2020.
7/doi. org/ 10. 1016/ 1tte:/10.1116/.aaee//e. aip.p...innn....... 2021. 01.
Kaixiong Zhou, Xiao Huang, Qingquan Song, Rui Chen, and Xia Hu. Auto-gnn: Neural architec- ture search of graph neural networks. Frontiers Big Data, 5, 2022. doi: 10.3389/FDATAA.022.. 1029307.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In 5th In- fernational Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. httt:ttps::/oott http:::///ppereview. net /et/Eozze wowormmmdd-rruBBoe..
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.I KNOWLEDGE GRAPH DEFINITION
knowledge graph (KG), denoted as G = {E, R. H. is structured representation of knowledge, where E' represents the set of entities, It the set of relations, and // the set of facts. Each fact h H is expressed as a triple (e1, r, E2). where e1, C2 E E are entities, and r E is a relation that connects them. These facts illustrate how entities linked by specific relations, forming graph structure that can be queried for relevant information. To incorporate more detailed information, entities and relations can have their own attributes, denoted Q, where Q {(gs, u²)] represents set of key- value pairs, with as the attribute type (key) and ey as the corresponding value. When relations are enriched with attributes, they are referred to hyper-relations and their facts become quadruples (e1, E2, Q).
A.2 ADDITIONAL RELATED WORKS
Another line of research focuses on leveraging existing datasct-model relationships to recommend models for unseen datasets (Wu et al. 2020 Tan et al. 2024). Different from our approach aimed at designing new models, these methods prioritize the application of pre-trained models, which are combined with their specifications to form learnware structured knowledge representation for future model recommendations. To enable effective matching between learnware and unseen datasets, (Wu et al.] 2020) employs a reduced kernel mean embedding method. More recently, Beimingwu (Tan et al. (0024) has integrated the learnware application pipeline into comprehensive platform, encompassing preparation, matching, and deployment processes.
A.3 KNOWLEDGE BENCHMARK GRAPH INFORMATION
Here we provide the examples about the structures of the Knowledge Benchmark Graph (KBG). We use the Nas- Pas--Bncch-rrape which is a benchmark dataset with performance records on 26,206 models for 9 graph datasets, to construct the KBG. The important entities are Data and Model, they are the have attributes for their detailed informaiton.
Data: as claimed in the main text, the D entity represents the dataset used for training and evaluating the models. As we are using the graph datasets, the numerical features are extracted from the graph structure by using the NetworkX library https::tts.. http///ewtortte [org/] In case the graph is too large to be processed, we randomly sample subgraphs of the original graph to calculate its statistics and note the corresponding attributes with 'local'. Fot other data modalities, their numerical features can be replaced by the corresponding statistical characteristics. The attributes of the Data entity include:
node_ fiatffeeeure the number node features.
edge feature: the number of edge features.
node.count: the number of nodes.
edge count: the number of edges.
num,classes: the number of node classes.
density: the density of the graph, calculated by the NetworkX library.
connected connectedd.coppoeettss components: other by paths.
the set of vertices in graph that linked to each
averageedegree: the average degree of all the nodes in the graph. average wusluusiii www.ooofiiiiieeemeemoomoo the degree to which nodes in a graph tend to cluster together.
average wwweaae..egreeeeooo.. dewreee.centralltt...... the average fraction of connected nodes across the whole graph.
the average influence of all nodes in the - local average clusteriiggooefffcciiiii coefficient: the average clustering coeffi- cient of the subgraphs.
average www.eveeeto..eentraaeeeeeeeeeeee
graph.
17

---

Published as a conference paper at ICLR 2025
"nare": "core",
"type": "dataset",
"property":
"node Teature": 1433,
"Loca Leverege__ ctrpes__clustrrnn_oefffeee coefficceef...
"aversge_clustering.cmm cesttrringcoeffccienn""." 9.24667229831193728,
"sverage_cegree_ceneee cettreltetraa'::: B. 1.9941999992299227777
"Local Laverage wtt.eeereeeeeeeeeneeeeeeeee
wwtweemess_ceentraaeeeeeeeeeeeee centtraltttt 0.0,
sarge_coutt::
"nun classes":
"density":: 9.90149999122992877,,
"average_ wwwpeeeeeeeorreeeeieeeeeeee sigewector_centraaltyy"""
4.84778545669887755,
couoee_ 2786,
raverage_cegrees
3.889/997000022285,
"comneceet_compnne
"lecal_graph_diaeeee"
"edge_featurd reature": 8,
"lacal_average.
wtt_aerge__hortteeee www.thhotttttteee patssttp__h________ patppath___egg Length":: 1.0,
"description":
dataset citation network 2,708 nachine-learninn papers, organized distinct
classes. These papers are interlinked by 5,429 citations, forsing directed graph that maps papers cite each other."
1,
"id":
Figure 7: The structure of the Data entity the KBG.
the average sum of the fraction of all all--pirss shortest paths that pass through node of the subgraphs.
Lvearave wwweeennesscentrralee
local_llee lerellgrraph.a Li ameter: the length of the shortest path between the most dis- lanced nodes of the subgraphs.
loca l_averageeeeeeeeeeeeeee www..ccoooeeesseennceee of the shortest path distances from a node to all other nodes of the subgraphs.
ity: the average reciprocal of the
description: the semantic description of the dataset.
The example shown in Fig.
Model: : the model contains the following attributes.:
num.post.layee the number of embedding layers before passing the raw node features into the GNN.
nun.prev.layers: the number of embedding layers after receving the hidden em- beddings from the GNN. dimension: the dimension of the hidden embeddings.
dropout: the drop out rate of the model.
Learning.rate: the learning rate of the model.
weight-decay: the weight decay during optimization.
num_epoch: the number of epochs of training the model.
And the example is shown in Fig.
As for the relations, the most important relation haspa f formance that records the performance
information of the KBG. An example is shown in Fig. 8b and its attributes include:
perf (acc.test:
loss: information the testing and validation accuracies and the loss on the test set is recorded in the form of dictionary.
acc_valid:
the performance
1 latency: the model training time.
para: the number of parameters of the optimized model.
task: the task for the corresponding performance record.
The entity and relation statistics of the extracted KBG is shown in Tab. and Tab. Please note that the search space Fe' contains 26,206 variants of model architectural designs, but considering the fine-grained pyraa-earrmeters of different models, we finally have 21 1,669 model instances in the
18

---

Published as a conference paper at ICLR 2025
"source_entity":
"target_entity"; 17,
"name": "00", "type": "model",
"relation": "has_performance",
"property": {
"property": {
"perf": {
_aun__pre_aayer"'"" 0,
"acc_test"; 0.797,
"nun_post_layers"" 1,
"acc__alid"; 0. 9.7733393333333334
" "dimension": 256,
"loss": 1.234
"dropout": 0.7,
} "latency": 8. 0.005922238832222311,,
setarningrrate", 0.1,
"weight :biight_deca";::: 0. 0.0005,
"para": 1.736711, "task": "Node Classification"
" "num_epoch": 400
b
"id":
"id": 2328395
(a) example model entity.
(b) The example hasPerformance relation.
Figure 8: Effect of performance threshold on A and performance.
Table 6: The entity statistics of KBG.
Entity Category
Entity Type
Num Instances
Dataset
Dataset
Domain
4
Heterogeneity
Model
21 11,669
Optimizer
GPU
2
Model
CPU
OS
2
Structure Topology
Layer Structure
9
Task
Common
Modality
Total Number
12
211,712
KBG, which leads to the increased number facts. However, in the actual model design retrieval, only the architectural design is considered, thus the valid number of models is still 26,206.
Subsequent to Nas-Bench-Graph, many methods are developed to improve the GNN design ef- fectiveness via proposing more effective search space. Early efforts explicitly incorporate link information into designing (Di et al. 2021 WANG et al. 2021). enhancing node classification and link prediction. Subsequent studies introduced fine-tuning search spaces, improving pre- pre-trained GNN adaptation (WANG et al.) 2024), while others developed data-adaptive GNNs that dynamically adjust receptive fields based on graph properties (Wang et al. 20236). There are also AutoGNN methods explored message- passing function search on knowledge graphs, enhancing model expres- siveness but remaining constrained to specific KG structures (Di & Chen (003). As the search space design is an orthogonal task to our work, we do not compare our method with these methods in the main text and stick to Nas-Bench-Gres
A.4 BENCHMARK DATA INFORMATION
The datasets used in the benchmark data are: Cora Sen et al. 2008). Citeseer (Sen et al. 2008) Pubmed (Sen et al., 2008). CS (Shchur et al.) 2018), Physics Shhchur al. 2018), Photo Shhhhr et al. 2018), Computers (Shchur et al. 2018). Arxiv (Hu et al. 2020), and Proteins (Hu et al.] 2020
The statistics of the datasets shown in Tab. 8 On the other hand, the models used in
19

---

Published as a conference paper at ICLR 2025
Table 7:
The relation statistics of KBG.
Relation Category
Relation Type
Num Instances
hasDomain
hasHeterogeneity
Dataset
hasTask
hasModality
hasOptimizer
211,669
hasGPU
211,669
hasCPU
211,669
hasOS
211.669
hasArchGraph
211.669
Model
hasLayerOperator 1
211,669
haslaayerOperator
211,669
hasLayerrperrator..
211,669
hasLayerOperator.
211,669
hasTask
211,669
has.oodality
211,669
Dataset-Model
hasPerformance
211,669
Total Number
16
2,540,064
the benchmark data have unique topologies with nodes in the directed acyclic graph (DAG) and 9 different layer operations, ['GCN", 'GAT', 'SAGE', "Skip', 'GIN", 'Cheb', 'FC', 'ARMA, "Graph"], ], each node.
Table 8: List of datasets used in the benchmark data.
Dataset
Nodes
Edges
Classes
Description
Cora
2,708
5,429
7
Citation
Citeseer
3,327
4,732
6
Citation
Pubmed
19,717
44,338
3
Citation
CS
18,333
81,894
15
Coauthor
Physics
34,493
247.962
5
Coauthor
Photo
7,487
119,043
8
Shopping
Computers
13,381
245,778
10
Social
Arxiv
169,343
1,166,243
40
Citation
Proteins
132,534
39,561,252
112
Protein
A.5 FULL EXPERIMENTS THE MODEL RETRIEVAL SCORE
The full results of the Tab. 2 with standard deviation is provided in Tab. 9 Because the similarity based retrieval methods always find the same model across different runs, the standard deviation is not reported.
A.6 ADDITIONAL EXPERIMENTS ON THE IMPACT OF A
In addition to the results in Sec. 7.3 we provide the case studies on the other two datasets Cora and Physics in Fig. and Fig. 10, Across the three datasets, e, which further validates the observation in Sec. [7. A higher 7 will generally result higher transferred performance all three datasets, which is also supportive
observe increasing trend of A with
observation.
20

---

Published as a conference paper at ICLR 2025
Table The performance of different bascline methods the datasets used with standard deviation.
Model
Cora
Citeseer
PubMed
CS
Physics
Photo
Computer
arXiv
GCN
80.97=0.39 69,90+1
1.26 77.4640.6 1026..26
88. 38.65=0.55 57 90.85
20 89. 39.4440.44
13.1640.55 71.08=0. 106
GAT
80,83+0.47 70.70+0.71 75.93=0.26
89. 19.47=1.14 78.7220.73
89.93+1.75 81.35=1.26 71.24=0.10
SAGE
79.47+0.31
66. 66.13+0.90 75.50+ 14 87.81 91.43+0.22 .9
88.29*1.03
81.46+0.73
70.78=0.17
GIN
79.77+0.38 63,30+1.22 1.26 76.74=0.80
81.08=3.09 86.67+0.88
87.37 73.95+0.
61.33+0,7 70
ChebNet
79,40=0.57 67. 37.03=1.. .02 75.13+0.44 89, 39.50=0.36 89, 19.75+0.87 86.65#0.77 79.10:2.26 70,87=0.1
ARMA
78.33+0.69
66. 66.20#0.75 75.00=0.51 89.87=0.35 88.8811 09 86.5543.35 78.47=0.57 70.87=0.1 17
k-GNN
78.06+0.47 30.97+3.56 75.38=0.97
83.81=0.58
88.9840.54
86.45+0.21
76.31=1.34
63.18=0.38
Random
77.87+2.41 66.64+1.33 1.32 74.
81.78
90.59#0.94 89.0412.55 76.61+3.56 68.93+1.8 1.82
EA
78.23+1.04 66.40+2.63
72.88+2.11 87.03=2.64
88.0722441
87330= 77.56+6.42 68,2822.95
RI
73.44+8.11 65.35+2.40
75.44+1.24
86. 16.1775.00
88. 18.1544.24
89.48+1.35
77.70=3.07 68.00=4.71
GNAS
78.55+1.20 63.25+5.87
73.04
86.04.7.85
89.54
87.27+2.96 70.96+9.66 1.711
Auto-GNN
78. 78.58+2.18 65.60+2.69
76.07=0.77
89.06=0.42 89.26+1.5 .51 1973 77.49=3.41 67.62#1.77
GPT4GNAS
78. 78.50=0.37 67.46+0..
73.89=0.86
89. 89.26=0.38
89,44 .94 89. 12+2.26 77.21 68. 98 1.222
GHINNS
79.13=0..45 67.35#0.44
74.90=0.57
89. 15 88.9442.57 89.42+1.99 77.0443.96 69.66= 1621..28
DesiGNN
80.31 69.20+0.16 76.60=0.00
89. 99.6420.08
10=0000
91.19=0.00 82.20=0.00 71.50=0.00
Kendall
67.73
69.20
71.80
88.56
91.56
88,90
76.85
71.49
Overlap
79.36
67.30
71.80
88.56
89.95
90.37
76.85
71.68
Ours
82.53
69.20
76.53
89.32
90.34
90.37
76.61
71.68
cora
physics
aniiv
T-
--
T=C 9B0
1-0 959
T:0
T-0 992
995
T:0 985
T-0.992
1-0,952
T=0.905
T-0.990
0.02
0.04
0.05
0.05
0.10
0.02
0.04
0.06
0.08
0.10
0.02
0.04
0.06
0.08
0.10
Figure 9: Effect of and A.
A.7 COMPLEX DATASET SIMILARITY COMPARISON
In this section, extend our experiment different dataset similarity metrics. In the main text, we compared several simple simialrities, and here we propose include more complex simi- larity metric, the Personalized PageRank (PPR) algorithm, further evaluate the dataset similarity metrics.
When implementing PPR, the original similarities S(D*, D calculated by the dataset features serve as the initial weights of edges. Considering that the unseen dataset 4 do not have model performance information, we neglect the performance relations between 1E'h and benchmark models M as they are less helpful for dataset similarity.
cora
physics
anxi
T=0,900
0.915
tw0.500
1=0.980
Pemorrance
0.815
TSC 992
1-0.00 989
0.305
0
0.9.69
Perfommnnc sormenne
0.710
T=0.992
1-0.969 06.955
0.810
C- 995
-- 0.595
0.712
0-9.905
0.805
0.855
0.800
0.590
0.02
0.04
2.06
0.08
0.10
0.02
0.04
0.06
08
0.10
0.02
0.24
0.05
0.10
Figure 10: Effect of and performance.
21

---

Published as a conference paper at ICLR 2025
Table 10: The RLS of different similarity metrics in initial model suggestion.
Metric
Cora
Citeseer
PubMed
CS
Phys.
Photo
Comp.
Arxiv
Simple-L1 0.452
0.126
0.003
0.020
0.013
0.310
0.025
0.014
Simple-T.2
0.498
0.064
0.012
0.007
0.040
0.328
0.036
0.012
Simple-Ours
0,435
0.340
0.004
0.025
0.002
0.252
0.043
0.027
Complex-PPR
0.853
0.768
0.278
0.340
0.344
0.316
0.485
0.493
The experiment results are shown in Tab 10 higher RLS indicates that the dataset similarity metric can better find a similar dataset. The best values are shown in bold and second best in underline.
Results show that, our metric obtains better performance than other naive metries for its surpres- sion on the overly large feature differences. When further considering the edge information among datasets, the PPR algorithm sigmificantly improves the dataset similarity quality over the naive meth- ods. This further validates the essence of constructing the knowledge benchmark graph. However, the linearity is still not high enough (maximally I), indicating that there still room for improve- ment in the dataset similarity metric design.
A.8 DETAILS IN COMBINING KBG WITH LLM
When combining KBG and LLM to suggest model designs, we position LLM as a model design optimization tool that could understand and leverage the information retrieved from KBG. As are agnoistic to the LLM designs, we use the GPT-4 model as the representative LLM in our exper- iments. The detailed procedure follows:
1. Given KBG and unseen dataset D", find similar datasets D D with dataset similarity score S(-,))
2. Retrieve promising candidate models I with the model relevance score R(-) based on D
3. Provide information of D Do, F and historical results and instruct the LLM to infer (design a new model based on F9) select (pick the most suitable model from FF) model F" for D
4. Evaluate and record the performance of F" D
5. Retrieve another of Ie from KBG based on: 1) relevance R(-) to D and 2) similar model architectures to F'.
6. Repeat steps 3-5 until the computation budget exhausted or the performance of satisfactory.
When retrieving the candidate models IF the step 5, combine the model relevance score R(-) with the model archietectural similarity (noted as W(-)) to find more suitable models. To calculate W(.). we first transform the topology and layer operations of the benchmark models into binary vectors w of Os and 1s. Each element of w stands for part of arthitecture design, and means that benchmark models have the same architecture the current model design F'. Then, average the elements in w as the model architecture similarity W(F, F") for J' E 1t F of F'. Finally, we use (R(F; D + (1 (I,(F, F") as the final model relevance for selecting next of candidates, where /3 is the weight parameee and set to 0.8 in our experiments.
When designing the prompts for LLMs, we adopt an intuitive and effective prompting practice with- out excessive tuning because we aim to validate the effectiveness of our KBG as extra knowledge assisting L.LMs design models. Following the prior examples in Wang al, (2023a): Dong et al. 2023): ; Wang et al. (2024), the prompts are organized into five textual components:
1. Task description
2. Model space description
3. Optimization trajectory
4. Candidate models from KBG
5. Role-play instruction for Infer/Select strategies
22

---

Published as a conference paper at ICLR 2025
These textual segments updated (specifically, components (3) and (4)) and combined into single prompt template during each iteration before sending to GPT-4 API for the response. Especially, when receiving the response from LLM, require the LLM to provide the structure topology and layer operations of the recommended model, along with the recommendation reason.
Afterwards, extract the structure topology and layer operations build the model automatically. Then, the recommended model is tested on the unseen dataset to obtain the performance feedback, which will be further combined with the model details and appended the optimization history in the prompt for the next iteration. More importantly, the recommended model will serve as the anchor for retrieving the next round of similar models from KBG, thereby improving the model performance over time.
Thoughout the process, the only constraint applied LLMs is that they camnot recommend model already tested in the optimization history, which can be enforced by proper instruction. Please note that our KBG can capture the similarity between datasets and models, which is fundamentally filtering mechanism that only retrieves the most relevant and effective knowledge before sending them to LLMs for reference. Therefore, we did not apply further filtering to LLM's suggestions so that the effectiveness of our KBG in this study could be directly observed.
A.9 CASE STUDY MODEL RETRIEVAL ON THE KBG
To help understand our process, we first breifly introduce the pipeline of retrieving initial model designs from KBG for unseen dataset below.
1. Given KBG with beenhmark datasets ID' and unseen dataset D we calculate the dataset similarity score S(D", D between D and each D E D based on their statistical fea- tures.
2. Select the similar datasets DY (S(D), D") 8|D E mD} D" up to threshold 6.
3. Calculate the relevance score RR-)) of the benchmark models F& towards D based on their performances on Dr
4. Select the candidate models F {R(F*) 2 F| F²} for D" up a threshold Recommend F to D
5.
For study, suppose dataset is the unseen dataset D" the more concrete steps
1. Calculate the dataset similarity between and other benchmark datasets, whose similar- ities are listed in Tab. [
2. Then set the threshold 8 to 0.7 and obtain the candidate datasets DD = {Comp.,
3. Calculate the model relevance R(F; D*) of the benchmark models towards cora based on their performances on Dh {computers, photo). Suppose model F has the reletiave performance 0.8 on computers and 0.7 on photo, then
R(F; D 0.8 x 0.735902 + 0.7 x 0.724709 1.095.
4. Select models with relevance score higher than
5. Recommend the selected models cora.
Table 11: The similarity between cora and other benchmark datasets.
Dataset
Comp.
Photo
Citeseer
Pubmed
CS
Phys.
arxiv
Similarity
0.735902
0.724709
0.69855
0.698388
0.688489
0.665778
0.647834
A. EXPERIMENTS ON APPLYING KBG TO IMAGE CLASSIFICATION BENCHMARK
In the main experiments, we demonstrated the effectiveness our framework in the graph domain. In this section, further validate its generalization ability by applying it to the computer vision
23

---

Published as a conference paper at ICLR 2025
domain. This demonstrates that our framework is extensible across different data modalities and model designs.
This generalizability is achieved through comprchensive ontology, which specifies how orga- nize dataset features, model architectures, and performance information into the Knowledge Bench- mark Graph (KBG) without relying domain-specifie knowledge. Additionally, both dataset similarity metric and model relevance metric depend solely on the constructed KBG, ensuring the framework's s adaptability across domains.
We conduct experiments on the NATS- Benchh dataset (Dong et al. 2021). a benchmark in the com- puter vision domain for image classification tasks. NATS-Bench contains performance records for three image datasets and 15,645 unique topological architectures.
Dataset Features and KBG Construction For this experiment, we extract features of the image datasets by transforming the statistical information of raw images into vectors and summarizing each statistical feature across samples. The remaining construction procedures follow our standard framework.
Using the NATS-Bench data and our KBG ontology, we constructed CV-KBG (Knowledge Bench- mark Graph for Computer Vision). The CV-KBG was then used to calculate dataset similarity scores as defined in Sec. S The results are shown Tab.
Table 12:
Dataset similarity scores for the datasets ImageNet16-120, Cifarl0 and Cifar100 that are used
ImageNet16-120
Cifar10
Cifar100
ImageNeet6-1221
1.000000
0.177472
0.31
Cifar10
0.177472
1.000000
0.510925
Cifar100
0.311604
0.510925
1.000000
The results align with intuitive expectations: CIFAR-10 and CIFAR-100 are closely related, while ImageNet16-122 is more similar to CIFAR-100 than CIFAR-10, reflecting the greater diversity of image classes in CIFAR-100.
Model Retrieval for Unseen Datasets Next, we simulated unseen datasets by excluding one dataset and retrieving the best models from the remaining similar datasets for the unseen dataset, whose results are shown in Tab. 13
Comparison between optimal and retrieved models for unseen datasets. The test accuracy of the retrieved models was compared with the accuracy of the optimal models, with rankings of the retrieved models shown in parentheses.
Table 13:
ImageNee16-120
Cifar10
Cifar100
Optimal
38.27 38.27 (1/15625)
89.16
61.18
Retrieved
88.96 (6/15625) 60.44 (14/15625)
These results validate that our retrieval method is effective in identifying high-performing models for unseen datasets in the computer visions domain. This demonstrates the generalization ability and practical utility of our framework.
24


Reviews:
[{"review_id": "tkozQ0mpRu", "reviewer": "Reviewer_NmMt", "summary": "This paper introduces a graph dataset that helps connect datasets, models, and model performance, making it easier for machine learning systems to automatically find the best model architecture for a specific dataset. Since real-world datasets are often new and unseen, the authors create a method to measure how relevant different datasets are to each other, which helps in sharing knowledge between them. This method allows the system to use information from existing benchmark data, ensuring that high-performing models can still be applied to new datasets. Additionally, the authors present a new metric that focuses on the most useful insights, which makes the model selection process even better. In their experiments, they test this approach on various datasets to show how effective and efficient it is, highlighting its potential to improve model design and performance in real-world situations.", "questions": "please refer to weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has several notable strengths that enhance its contribution to the field of automated machine learning.\n\n1. The introduction of a comprehensive graph dataset models the relationships between datasets, models, and performance. This structured resource simplifies the model selection process for researchers and practitioners.\n2. The theoretical framework is well-articulated and provides a solid basis for the proposed methods. This enhances the credibility of the approach and demonstrates a deep understanding of the principles involved.\n3. The experiments conducted are thorough and well-executed, testing the methods across various datasets. These results provide strong empirical support for the authors’ theoretical claims.\n4. The research has significant implications for automated machine learning (AutoML), allowing for the automatic identification of optimal model architectures. This capability can reduce the time and expertise required for model design, making machine learning more accessible.\n\nOverall, the paper effectively combines a valuable dataset, strong theoretical foundations, and solid experimental validation, positioning it as a promising contribution to AutoML. Its findings could lead to further advancements in automated processes for model development.", "weaknesses": "The theoretical explanations in the paper could be improved with additional background to aid reader comprehension. \n1. For example, in Definition 1, it would be useful for the authors to provide an overview of existing problem formulations in model transfer or AutoML to better contextualize their approach. \n2. Explaining the motivation for using a probability lower bound in Definition 1 and its relevance to practical model transfer would clarify this choice. \n3. It would also be helpful to indicate whether this problem formulation is novel or based on existing methods, and if it is novel, to discuss the advantages it brings over previous formulations.\n\nIn Section 4.3, the intuition behind the transferability score could be further clarified. \n1. A conceptual explanation of what the transferability score represents in practical terms would be beneficial, along with a small example or illustration to demonstrate how it is calculated and interpreted. \n2. Additionally, comparing this score with existing metrics for evaluating model transfer effectiveness could further clarify its utility.\n\nFurthermore, the paper’s discussion on integrating Large Language Models (LLMs) into the proposed framework could be more comprehensive, as it is currently quite brief. \n1. The authors might expand on the specific role of LLMs in their approach, detailing how they interact with the Knowledge Benchmark Graph and contribute to model selection or adaptation. \n2. Examples illustrating the LLMs' role in the process would be helpful, as well as a discussion of any potential challenges in integrating LLMs and how these are addressed. \n3. Lastly, comparing this approach to other recent methods that incorporate LLMs for AutoML or model selection would provide a useful context for the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gEBBwrH0H4", "reviewer": "Reviewer_zioe", "summary": "The authors proposes a new solution for AutoML. The authors design a knowledge graph that contains information on data, model and performance. With the knowledge graph, the method uses similarity score of data and relevance score of model to suggest best model on unseen data. \n\nIn experiments, the authors construct the knowledge graph with graph datasets and GNN architectures. The result show that the method author proposes achieves the best result on 3/8 tasks. However, with the assistance of LLM, the model is able to achieve best result on 5/8 tasks.", "questions": "1. LLM assisted method achieves best performance but there is no detailed explanation or design. For example, you can answer the following questions:\n    (a)What prompts or instructions were given to the LLM?\n    (b)How were the LLM's outputs processed or integrated into the model selection process?\n    (c)Were there any constraints or filtering applied to the LLM's suggestions?\n2. It's good to include more complicated design of the two scores. For example, incorporate edge-level information in data similarity score calculation and model architectural information, like the number of convolutional layers, in model relevance calculation.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper writing is clear. The formulation is very straight-forward. The authors use data similarity score and model relevance score to infer the best potential model on unseen data. \nThe authors provide extensive experiments comparing to SOTA methods.", "weaknesses": "1. The data similarity score the authors propose is too simple. The authors focus their study on GNN, but the similarity score does not involve any relational information on the edges. \n2. The model relevance score is confusing. The name sounds like it look for architectural similarity between models. But in reality it's the model's historical performance.\n2. The experiment section that involves LLM is very vague to me. There's is not explanation on exactly how LLM infer or select the models.\n\nIn summary, the method basically proposes models based on the following two ideas: a) if the node features are similar 2) if the model has historical performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WNfhTuJu6l", "reviewer": "Reviewer_eVNQ", "summary": "This paper attempts to construct a graph that stores all the existing datasets, models, and model performance on datasets for future research and development endeavors. Based on the constructed datasets, the authors try to propose some metrics to evaluate the similarity between datasets and the effectiveness of the models retrieved on an unseen dataset. The experimental results show that such a graph is benefical for the development of AutoML.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is interesting and the motivation is convincing. \nThe implementation of this idea is relatively complete, including the graph construction process, the design of enhancing generalization ability on incorporating unseen datasets, and the retrieval mechanism of existing model candidates.", "weaknesses": "The work is full of engingeering skills while lack some academic insights. For example, controling and varying the hyperparameters (e.g., delta, the size of datasets, epsilon, etc.) bring limited insight. Perhaps a case study is required to illustrate how the algorithm succeeds to retrieve a good model according to a given unseen-yet-similar dataset. There should be more deeper insights and factors beyond the similarity of datasets, such as the underlying common research issues. What features should the algorithm capture and consider? \nThe scenario is relatively limited. Authors conduct experiments merely in GNN domain. It’s unclear whether such an effort could generlize to other ML methods, which makes the contribution of this paper vague. I suggest conducting a small amount of experimental evidence to demonstrate the generalization ability of this work, which could make it more promising and convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BmdhILf6RQ", "reviewer": "Reviewer_JG3d", "summary": "- This paper proposes a novel AutoML framework with LLM for GNN design. In this paper, the authors construct a knowledge benchmark graph to inform the LLM with more domain knowledge about GNN architecture and design new metrics to guide the knowledge selection.", "questions": "- Which LLM do you employ in the experiments? I can not get the corresponding information after reading Section 5.1\n- The naming of knowledge benchmark graph might be modified. I think the current name misleads people into thinking this is a new KG benchmark.\n- See Weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of this paper of paper is more innovative, combining Knowledge Grap, LLM and AutoML.\n- The authors have done sufficient data preparation, method design and experiments around this idea.", "weaknesses": "- There are some labeling errors in Table2, e.g., the optimal result on the Citeseer dataset appears on the GAT but is not bold.\n- The experiments in the current article stop at the GNN domain and are only oriented to the node classification task, which has some limitations in the scope of application. And the title gives the impression that the authors' approach is oriented to a variety of tasks in generalized scenarios. I think KBG can be considered to be applied on top of more heterogeneous tasks, such as other tasks in the field of graph learning, or even out-of-domain experiments such as CV/NLP that require the GNN method to verify the effectiveness of the method. Further, the design of the model can also be not limited to the GNN model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)