Structuring Benchmark into Knowledge Graphs to Assist Large Language Models in Retrieving and Designing Models

Authors: Hanmo Liu, Shimin Di, Jialiang Wang, Zhili Wang, Jiachuan Wang, Xiaofang Zhou, Lei Chen
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Knowledge Graph, Auto Machine Learning

Abstract:
In recent years, the design and transfer of neural network models have been widely studied due to their exceptional performance and capabilities. However, the complex nature of datasets and the vast architecture space pose significant challenges for both manual and automated algorithms in creating high-performance models. Inspired by researchers who design, train, and document the performance of various models across different datasets, this paper introduces a novel schema that transforms the benchmark data into a Knowledge Benchmark Graph (KBG), which primarily stores the facts in the form of performance(data, model). Constructing the KBG facilitates the structured storage of design knowledge, aiding subsequent model design and transfer. However, it is a non-trivial task to retrieve or design suitable neural networks based on the KBG, as real-world data are often off the records. To tackle this challenge, we propose transferring existing models stored in KBG by establishing correlations between unseen and previously seen datasets. Given that measuring dataset similarity is a complex and open-ended issue, we explore the potential for evaluating the correctness of the similarity function. Then, we further integrate the KBG with Large Language Models (LLMs), assisting LLMs to think and retrieve existing model knowledge in a manner akin to humans when designing or transferring models. We demonstrate our method specifically in the context of Graph Neural Network (GNN) architecture design, constructing a KBG (with 26,206 models, 211,669 performance records, and 2,540,064 facts) and validating the effectiveness of leveraging the KBG to promote GNN architecture design.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1729868943977

thread_type:
official_review

<thread_object>
creator: reviewer_jg3d

<content>
summary:
- This paper proposes a novel AutoML framework with LLM for GNN design. In this paper, the authors construct a knowledge benchmark graph to inform the LLM with more domain knowledge about GNN architecture and design new metrics to guide the knowledge selection.

strengths:
- The idea of this paper of paper is more innovative, combining Knowledge Grap, LLM and AutoML.
- The authors have done sufficient data preparation, method design and experiments around this idea.

weaknesses:
- There are some labeling errors in Table2, e.g., the optimal result on the Citeseer dataset appears on the GAT but is not bold.
- The experiments in the current article stop at the GNN domain and are only oriented to the node classification task, which has some limitations in the scope of application. And the title gives the impression that the authors' approach is oriented to a variety of tasks in generalized scenarios. I think KBG can be considered to be applied on top of more heterogeneous tasks, such as other tasks in the field of graph learning, or even out-of-domain experiments such as CV/NLP that require the GNN method to verify the effectiveness of the method. Further, the design of the model can also be not limited to the GNN model.

questions:
- Which LLM do you employ in the experiments? I can not get the corresponding information after reading Section 5.1
- The naming of knowledge benchmark graph might be modified. I think the current name misleads people into thinking this is a new KG benchmark.
- See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533058993

thread_type:
official_review

<thread_object>
creator: reviewer_evnq

<content>
summary:
This paper attempts to construct a graph that stores all the existing datasets, models, and model performance on datasets for future research and development endeavors. Based on the constructed datasets, the authors try to propose some metrics to evaluate the similarity between datasets and the effectiveness of the models retrieved on an unseen dataset. The experimental results show that such a graph is benefical for the development of AutoML.

strengths:
The idea is interesting and the motivation is convincing. 
The implementation of this idea is relatively complete, including the graph construction process, the design of enhancing generalization ability on incorporating unseen datasets, and the retrieval mechanism of existing model candidates.

weaknesses:
The work is full of engingeering skills while lack some academic insights. For example, controling and varying the hyperparameters (e.g., delta, the size of datasets, epsilon, etc.) bring limited insight. Perhaps a case study is required to illustrate how the algorithm succeeds to retrieve a good model according to a given unseen-yet-similar dataset. There should be more deeper insights and factors beyond the similarity of datasets, such as the underlying common research issues. What features should the algorithm capture and consider? 
The scenario is relatively limited. Authors conduct experiments merely in GNN domain. It’s unclear whether such an effort could generlize to other ML methods, which makes the contribution of this paper vague. I suggest conducting a small amount of experimental evidence to demonstrate the generalization ability of this work, which could make it more promising and convincing.

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662526375

thread_type:
official_review

<thread_object>
creator: reviewer_zioe

<content>
summary:
The authors proposes a new solution for AutoML. The authors design a knowledge graph that contains information on data, model and performance. With the knowledge graph, the method uses similarity score of data and relevance score of model to suggest best model on unseen data. 

In experiments, the authors construct the knowledge graph with graph datasets and GNN architectures. The result show that the method author proposes achieves the best result on 3/8 tasks. However, with the assistance of LLM, the model is able to achieve best result on 5/8 tasks.

strengths:
The paper writing is clear. The formulation is very straight-forward. The authors use data similarity score and model relevance score to infer the best potential model on unseen data. 
The authors provide extensive experiments comparing to SOTA methods.

weaknesses:
1. The data similarity score the authors propose is too simple. The authors focus their study on GNN, but the similarity score does not involve any relational information on the edges. 
2. The model relevance score is confusing. The name sounds like it look for architectural similarity between models. But in reality it's the model's historical performance.
2. The experiment section that involves LLM is very vague to me. There's is not explanation on exactly how LLM infer or select the models.

In summary, the method basically proposes models based on the following two ideas: a) if the node features are similar 2) if the model has historical performance.

questions:
1. LLM assisted method achieves best performance but there is no detailed explanation or design. For example, you can answer the following questions:
    (a)What prompts or instructions were given to the LLM?
    (b)How were the LLM's outputs processed or integrated into the model selection process?
    (c)Were there any constraints or filtering applied to the LLM's suggestions?
2. It's good to include more complicated design of the two scores. For example, incorporate edge-level information in data similarity score calculation and model architectural information, like the number of convolutional layers, in model relevance calculation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696277209

thread_type:
official_review

<thread_object>
creator: reviewer_nmmt

<content>
summary:
This paper introduces a graph dataset that helps connect datasets, models, and model performance, making it easier for machine learning systems to automatically find the best model architecture for a specific dataset. Since real-world datasets are often new and unseen, the authors create a method to measure how relevant different datasets are to each other, which helps in sharing knowledge between them. This method allows the system to use information from existing benchmark data, ensuring that high-performing models can still be applied to new datasets. Additionally, the authors present a new metric that focuses on the most useful insights, which makes the model selection process even better. In their experiments, they test this approach on various datasets to show how effective and efficient it is, highlighting its potential to improve model design and performance in real-world situations.

strengths:
This paper has several notable strengths that enhance its contribution to the field of automated machine learning.

1. The introduction of a comprehensive graph dataset models the relationships between datasets, models, and performance. This structured resource simplifies the model selection process for researchers and practitioners.
2. The theoretical framework is well-articulated and provides a solid basis for the proposed methods. This enhances the credibility of the approach and demonstrates a deep understanding of the principles involved.
3. The experiments conducted are thorough and well-executed, testing the methods across various datasets. These results provide strong empirical support for the authors’ theoretical claims.
4. The research has significant implications for automated machine learning (AutoML), allowing for the automatic identification of optimal model architectures. This capability can reduce the time and expertise required for model design, making machine learning more accessible.

Overall, the paper effectively combines a valuable dataset, strong theoretical foundations, and solid experimental validation, positioning it as a promising contribution to AutoML. Its findings could lead to further advancements in automated processes for model development.

weaknesses:
The theoretical explanations in the paper could be improved with additional background to aid reader comprehension. 
1. For example, in Definition 1, it would be useful for the authors to provide an overview of existing problem formulations in model transfer or AutoML to better contextualize their approach. 
2. Explaining the motivation for using a probability lower bound in Definition 1 and its relevance to practical model transfer would clarify this choice. 
3. It would also be helpful to indicate whether this problem formulation is novel or based on existing methods, and if it is novel, to discuss the advantages it brings over previous formulations.

In Section 4.3, the intuition behind the transferability score could be further clarified. 
1. A conceptual explanation of what the transferability score represents in practical terms would be beneficial, along with a small example or illustration to demonstrate how it is calculated and interpreted. 
2. Additionally, comparing this score with existing metrics for evaluating model transfer effectiveness could further clarify its utility.

Furthermore, the paper’s discussion on integrating Large Language Models (LLMs) into the proposed framework could be more comprehensive, as it is currently quite brief. 
1. The authors might expand on the specific role of LLMs in their approach, detailing how they interact with the Knowledge Benchmark Graph and contribute to model selection or adaptation. 
2. Examples illustrating the LLMs' role in the process would be helpful, as well as a discussion of any potential challenges in integrating LLMs and how these are addressed. 
3. Lastly, comparing this approach to other recent methods that incorporate LLMs for AutoML or model selection would provide a useful context for the reader.

questions:
please refer to weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732500580957

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to express our heartfelt thanks to all reviewers (**NmMt**: Rating 5; **zioe**: Rating 5; **eVNQ**: Rating 6; **JG3d**: Rating 8) for their insightful comments and valuable feedback on our submission! 
We are particularly grateful for the recognition of our work’s strengths, including its novel and convincing approach to retrieving models for unseen datasets without training, which advances research in AutoML (**NmMt**, **eVNQ**, **JG3d**); its clear and well-articulated theory for evaluating dataset similarity metrics (**NmMt**, **zioe**), supported by a comprehensive method design for retrieving datasets and models (**eVNQ**, **JG3d**); and the solid experimental validation of our idea (**NmMt**, **zioe**).

Regarding the concerns raised by the reviewers, we believe we have provided comprehensive and satisfactory responses:

**C1**: Certain definitions and scoring metrics require more intuitive explanations.

To tackle this, we have supplemented the paper with background information, intuitive explanations, and examples in the reply series of **Reply to NmMt-W1** and **Reply to NmMt-W2**. 
The revised submission includes an expanded background in **Sec. 2**, an intuitive problem definition and a detailed comparison with AutoML and model transfer in **Sec. 3**, and a clearer description of the transferability score in **Sec. 6**.

---

**C2**: It is encouraged to include more complex dataset similarity and model relevance scoring metrics.

We have explained our reasons for choosing the current metrics in **Reply to zioe-W1** and **Reply to zioe-W2**, and provided our ideas and experiments on more complex metrics designs in **Reply to zioe-Q2**.

---

**C3**: Lack of details in how the LLM is combined with the knowledge benchmark graph (KBG).

We provided the pipeline of our LLM+KBG method in **Reply to NmMt-W3**, **Reply to zioe-W3**, and **Reply to JG3d-Q1**. In particular, we provided additional details on the role of the LLM in model recommendation in **Reply to NmMt-W3.2** and how we format the inputs/outputs for the LLM in **Reply to zioe-Q1**. We have clarified the experiment description and added more details of LLM usage in the revised submission, please check **Sec. 7.6** and **Appendix A.6**.

---

Additionally, we have corrected the typos in **Tab. 2** (**JG3d-W1**), highlighted our academic insights (**eVNQ-W1**) and provided a case study of our method in **Appendix A.7** (**eVNQ-W1.2**). We have noted all the paper revisions in blue for easy identification.

---

Finally, we sincerely thank the Area Chair (AC) and all reviewers for their efforts! We look forward to further discussions to help us improve the quality of this paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a novel framework called Knowledge Benchmark Graph (KBG) that aims to assist in automated model design by leveraging existing benchmark knowledge and Large Language Models (LLMs). The key scientific claims include: The KBG effectively captures dataset-model-performance relationships in a structured format that enables efficient retrieval and transfer of knowledge for unseen datasets. The proposed similarity metrics for datasets and model relevance enable effective model selection for new datasets by leveraging prior benchmark knowledge.

The evaluation focuses exclusively on GNN architectures and node classification. These experiments-based applications are currently difficult to attract board researchers. There is no demonstration of generalizability to other domains or tasks. There are limited theoretical insights beyond engineering aspects.

Decision: Accept

Primary reasons for rejection:
The reviewers generally agree that the paper presents strong experimental validation.The paper successfully combines knowledge graphs, LLMs, and AutoML in a coherent and practical framework for model design. The paper is well-written with straightforward formulations and clear methodology descriptions.