AutoG: Towards automatic graph construction from tabular data
Authors: Zhikai Chen
Han Xie
Jian Zhang
Xiang song
Jiliang Tang
Huzefa Rangwala
George Karypis
Venue: ICLR 2025 Poster (2025)
Keywords: Graph machine learning
Automatic data science
Applications of large language models
Tabular data
Primary area: learning on graphs and other geometries & topologies

Abstract: Recent years have witnessed significant advancements in graph machine learning (GML), with its applications spanning numerous domains. However, the focus of GML has predominantly been on developing powerful models, often overlooking a crucial initial step: constructing suitable graphs from common data formats, such as tabular data.
This construction process is fundamental to applying graph-based models, yet it remains largely understudied and lacks formalization.
Our research aims to address this gap by formalizing the graph construction problem and proposing an effective solution. We identify two critical challenges to achieve this goal: 1. The absence of dedicated datasets to formalize and evaluate the effectiveness of graph construction methods, and 2. Existing automatic construction methods can only be applied to some specific cases, while tedious human engineering is required to generate high-quality graphs. 
To tackle these challenges, we present a two-fold contribution.
First, we introduce a set of datasets to formalize and evaluate graph construction methods. 
Second, we propose an LLM-based solution, AutoG, automatically generating high-quality graph schemas without human intervention.
The experimental results demonstrate that the quality of constructed graphs is critical to downstream task performance, and AutoG can generate high-quality graphs that rival those produced by human experts. Our code can be accessible from https://github.com/amazon-science/Automatic-Table-to-Graph-Generation.

Published as a conference paper at ICLR 2025
AUTOG: TOWARDS AUTOMATIC GRAPH CONSTRUC- TION FROM TABULAR DATA
Zhikai ; Han Xie?, Jian Zhang', Xiang Song", Jiliang Tang', Huzefa Rangwala', George Karypis²
Michigan State University 2 Ammzzon
ABSTRACT
Recent years have witnessed significant advancements in graph machine learning (GML), with its applications spanning numerous domains. However, the focus of GML has predominantly been developing powerful models, often overlook- ing a crucial initial step: constructing suitable graphs from common data formats, such as tabular data. This construction process is fundamental to applying graph- based models, yet it remains largely understudied and lacks formalization. Our research aims to address this gap by formalizing the graph construction problem and proposing an effective solution. We identify two critical challenges to achieve this goal: The absence of dedicated datasets to formalize and evaluate the ef- fectiveness of graph construction methods, and 2. Existing automatic construction methods can only be applied to some specific cases, while tedious human engi- neering is required to generate high-quality graphs. To tackle these challenges, we present a two-fold contribution. First, we introduce a set of datasets to formal- ize and evaluate graph construction methods. Second, we propose solution, AutoG, automatically generating high-quality graph schemas without human intervention. The experimental results demonstrate that the quality of constructed graphs is critical to downstream task performance, and AutoG can generate high-quality graphs that rival those produced by human experts. Our code can be accessible from /attps://githus com/ amazon scioonne/cecc//// www.ooantatteaaaa-----------eeae
LLM-based
Generattinnne
INTRODUCTION
Graph machine learning (GML) has attracted massive attention due to its wide application di- verse fields such as life science (Wong et al. 2023), E-commerce (Ying al. 2018), and social
networks (Wang & Kleinberg 2023: Suárez- Varela et al. 2022). GML typically involves applying
models like graph neural networks (GNNs) (Kipf & Welling 2017 Ma & Tang
2021) leverage
the underlying graph structure of a given task, e.g., using the friendship networks for user recom-
mendations (Tang al. 2013) and identifying new drug interactions (Zitnik et al., 2018).
Despite the widespread interest and rapid development in GML (Kipf & Welling 2017 Ma & Tang 2021, constructing graphs from common data formats such as industrial tabular data Ghosh al.
2018 remains an under-explored topic. This primarily stems from a widely adopted assumption that appropriate graph datasets exist for downstream tasks akin established benchmarks (Hu al. 2020a]|hatur Khatua et al. (202).. However, readily available graph datasets are absent in many roal-world enterprise scenarios. First, given an input data in common storage formats such as tables, there can be many plausible graph schemas and structures that can be defined over them. The choice of graph schema impacts downstream performance of GML. Rossi et al. (2024) shows that considering the directional aspect of edges within graph can lead to substantial variance in the downstream GML performance. Second, converting the source data into graph format requires expert data engineering and processing. Even though, GNN based approaches shows strong performance on Kaggle leader- board (Wang et al. 2024c), it involves laborious preprrcesssig and specialized skills to transform the original tabular data into ready-to-be-ed bede--to--e-oosumeeed graphs for GML.
"The work done while heing an intern at Amazon.

---

Published as a conference paper at ICLR 2025
The objective of this work is to formalize the challenges in graph construction by establishing real- world datasets followed by automatic graph construction from imput tabular data.
Existing tabular graph datasets such as Wang al. (2024c) and [eyy al. (2024) assume the availability of well-formatted graphs with explicit relationships such as complete foreign-key and primary-key pairs. In these cases, graphs can be easily constructed using heuristics like Row2Node (Cvitkovic) (020) by converting each table into a node type. However, implicit relation- ships like columns with similar semantics (Dong al. 2023) or columns with categorical types also widely exist in real-world scenarios, which cannot be addressed by heuristic methods (see Figure I Datasets designed for evaluating graph construction should reflect the importance of modeling im- plicit relationships. Additionally, different tasks can be defined based on the same dataset (Fey) et al. 2024). Further, different ways to construct graphs affect different tasks' performance is an understudied problem. Therefore, ideal datasets for graph construction should also include different downstream tasks to reflect this problem. From the solution perspective, graph construction involves finding the best candidate among all possible graph structures. However, considering the vast search space, finding the graph structure through an exhaustive search is infeasible. Therefore, an effective automatic graph construction method should be able to efficiently identify high-quality candidates from many possible graph structures/schemas.
To address the above challenges, we propose set of evaluation datasets and large language model (LLM))baaed graph construction solution. We first extract raw tabular datasets from Kaggle, Co- dalab, and other data sources to design datasets reflecting real-world graph construction challenges. They differ from prior work (Fey et al.) 2024 Wang et al., 2024c) in that these datasets haven't It been processed by experts, and existing graph construction methods get inferior performance (see Table Dp. To solve the graph construction problem, we propose an LLM-based automatic graph construction solution AutoG inspired by LLM's reasoning capability to serve as a planning module for agentic tasks (Zhou et al., (2244 and tabular data processing (Hollmann et al. 2023). How- ever, we observe that L.LMss tend to generate invalid graphs or graphs with fewer relationships (as shown in Section (.3.T). We address this problem by guiding LLMs to conduct close-ended func- tion calling (Schick et al., 2024). Specifically, we decompose the generation of graph structures into four basic transformations applied to tabular data: (1) establishing key relations between two columns (finding cross-table relationships), (2) expanding a specific column (finding self-induced relationships), (3) generating new tables based columns (data normalization), and (4) manipulat- ing primary keys (generating proper node and edge types). Coupled with chain-of-thought prompt demonstrations for each action, AutoG generates series of actions get the augmented schema and thus construct the graph. We further demonstrate the effectiveness of AutoG on our proposed benchmarks.
Our major contributions can be summarized as follows:
a) Formalizing graph construction problems with set of datasets: We create set of datasets covering diverse graph construction challenges, consisting of eight datasets from academic, E- commerce, medical, and other domains.
b) LLM-based automatic graph construction method: AutoG:
To solve the graph construction problem without manual data engineering, we propose an L--M-based baseline to automatically generate graph candidates and then select the best candidates efficiently.
c) Comprehensive evaluation: We compare AutoG with different baseline methods on the pro- posed datasets. AutoG shows promising performance that is close to the level of data engi- neering expert.
2 PRELIMINARIES
2.1 TABULAR DATA AND SCHEMAS
The input tabular data is represented using the RDB language (Codd) 2007 Chen 1976) as schema file. Subsequently, we introduce table schemas and how they may be used to describe graph. We start by introducing the fundamental elements of RDB languages.
Definition. Tabular data D contains an array of K tables D = (Ti) Each table T, can be
viewed as set T: (C,, R, M.), where
2

---

Published as a conference paper at ICLR 2025
C = (Ci,,,.... Ci.t)) is array of strings representing the column names, with I denoting the number of columns in Tr
R, is matrix where each row R (R..g...... ................................ R.,) contains the values for the j-th row of table T,
M, (M,,,-.... Me1,)) is array specifying the data type of each column.
In this paper, we consider the following data types fcategory, numeric, c, text, primary (ey(K))) foreign.key (FK), set, timestamp) As example, if M2,1 text, then all values in the same column 1 ... Ri, are of text type (m, refers to the number of rows for table T.). Detailed descriptions of each data type can be found Appendix A.I
The definitions above focus on the properties of individual tables, For multiple tables with K > 1, they can be related with of n PK-FK pairs {a yes: 2ig V2&) where m .,.... M.. and y represent the indices of tables in 'D and the indices of columns. In this paper, we consider the scenario that multiple tables are retrieved for a specific downstream task and no explicit key relationships are given (Gan et al. 2024).
Table schema and graph schema description. Based on this language, we define table schema by storing all the meta information in structured format like YAML. (Ben-Kiki et al. 2009). An exam- ple shown in Appendix AA2 Table schema defines the metainformation of tables in structured manner following the RDB language. Graph schema is special type of table schema. Compared to general table schema, graph schema presents tables with proper column designs and PK-FK rela- tions. These characteristics make it trivial to convert graph schema (as discussed in Section 2.2 into an ideal graph structure for downstream tasks.
2.2 BRIDGING TABULAR DATA AND GRAPHS
Based on the definition of tabular data, the goal of graph construction is to convert relational tabular data D into graph G. Following Fey et al. (2024); /Wan al.) (2024c), we consider & as hetero- geneous graph (Wang et al. 2022) G {V,E) characterized by sets of nodes V and edges E. The nodes and edges are organized such that U.Ev / y and set of nodes of type and e represents the set of edges of type The main challenge of graph construction lies in extracting appropriate node types and edge types from the schema of tabular data. This process could be straightforward if treat each table node type and each PK-FK relationship as an edge type. However, this method may generate suboptimal graphs for general ta- ble schemas. For instance, when two entities placed in single table, entity might be treated as a feature of the other, resulting in a graph that fails to effectively reflect structural relationships. thereby impacting the performance of downstream tasks (Wang et al. 2024c).
U.eg E where y represents the
TebGNN chellenges
Paper
only
PaperiD
Paper_Cite
CI: User- UseriD
Paper Cited
MovielD
Tays
Label
Title
Text
User
Cot
Genres
Sat
MavielD
FK
Paper
Cd: Proper type
(remove PK)
Tag
text
different graphs
Tasks reguire
Field
C2: Cal to
Ratings
HasGenre
relation
RatingID
MavielD
FK
Writes
Affiliated
UserlD
Cat
Cot
Author
Cat
AuthorName
Cat
MovielD
Paper
FK
Institution
Cat
Time
Time
Author-
(Augmented table)
AuthorName
(a) Movielens
(b) MAG
Figure 1: Demonstrations of challenges in two selected datasets. Existing heuristic-based methods cannot well tackle C2-C4 in that they require task-specific decisions.
3 DESIGNING DATASETS FOR EVALUATING GRAPH CONSTRUCTION
To make the graph construction problem concrete and provide a set of datasets for comparing differ- ent methods, we first identify key problems that need to be addressed during the graph construction
3

---

Published as a conference paper at ICLR 2025
process. Based these problems, we have carefully selected 8 multi-tabular datasets from diverse domains to construct datasets for graph construction.
3.1 DESIGN SPACE OF THE DATASETS
We first propose four core challenges to be addressed when converting tabular data into graphs. Examples of these challenges are demonstrated in Figure
1. C1: Identifying missing cross-table PK-EW/K-FK relationships: Traditional methods like Row2Node (Wang al. 2024c) only turn PK-FK relationships into edges, while these relation- ships are usually not complete, which necessitates either automatic join discovery (Dong et al.) 2023) or human intervention. Compared to traditional join discovery whose aim to merge tables together, the role of identifying cross-table PK-FK is to find proper edge relationships beneficial to the downstream tasks, which is more challenging.
2. C2: Identifying self-induced relationships: Beyond the relationship across different tables, tables may present relational connections with their columns. For example, the "Field" column in Figure can induce useful relations, and thus, augmented table should be added. This corresponds to the single FK problem discussed in Gan et al. (2024). Identifying such self- induced relationships has been shown be beneficial for tasks like recommendation (Wu al.) 2021).
C3: Transforming tables into proper node edge types: How to convert tables into ap- propriate types affects downstream task performance and the validity of generated graphs For instance, the "Writes" table in Figure shouuld be better modeled as an edge type since it contains two foreign keys, and records the citation relationship between papers in table "Paper".
C4: Generating proper graphs for different downstream tasks(*): Considering that multiple tasks can be defined based on the same tabular data (Fey et al.) (024). one single graph design may not fit all tasks. This claim has not been well studied and will be verified in the experiment. This task is the most challenging and may require substantial work.
Design philosophy of these challenges. These four challenges are inspired by existing works (Wang et al. 2024ct Dong al. 2023 Gan et al. 2024) but go beyond their scopes. Specifically, CT is common problem in data lakes and RDB Dong et al. 20033 Hulsebos et al. 2019) for automatic data engineering. When constructing the graph is the final objective, joinable column detection becomes even more important since it's crucial to find relations. original schema from Kaggle to the graph schema used in Wang al. (2024c). Human experts have introduced multiple augmented tables, which are crucial to the performance of GML models. The mechanism behind these augmented tables hasn't been well studied, and we first introduce them in our datasets. C3 is derived from real-world datasets such as (Harper & Konstan 2015). and we find that simple heuristics may work poorly when the proper type of table cannot induced from the schema. C4 is naturally derived from the multiple tasks defined on tabular data. We are the first to study the influence of graphs on different downstream task performance.
C2 is derived by comparing the
Relationship to traditional database profiling (Abedjan et al., 2015). Database profiling, includ- ing normalization, is a related concept to our work. The goal of graph construction from relational data to graph is to find what kind of relational information is beneficial to the downstream task. For example, the objective of challenge is to consider whether the relationship induced by this categorical value is beneficial. This decision needs to consider the semantic relationship between this column and the corresponding downstream tasks, which cannot be solved by normalization. As comparison, profiling aims minimize data redundancy and improve data integrity. Despite the overlap, data profiling method cannot fully solve the graph construction task. In Section 4 we also design two functions which can conduct data normalization, and we find that LLM itself plausible decision maker for data normalization.
3.2 DATASETS
Based on the design space of graph construction from relational tabular data, we gather 8 datasets from various domains to evaluate graph construction methods. We collect these datasets from 1. the source of existing tabular graph datasets, such Outbrain (Wang al. 2024c);; 2. augmented from existing tabular graph datasets, such as Stackexchange (Wang et al. (2024c); traditional tabular datasets adapted for graph construction, including IEFF-CIS (Howard al., 2019) and Movie-
4

---

Published as a conference paper at ICLR 2025
lens (Harper & Konstan 2015). The information of these 8 datasets is listed in Table | Two concrete examples are shown in Figure Details on dataset sources and pre- pre-processing are shown in Appendix B
Table Our proposed datasets. The tasks are categorized into predictions relation attribute, entity attribute, and FK by following (Wang al. 2024c).
Name of the dataset
FTanks
#Tables
Inductive
CI
C2
C3
C4
Task type
Source of datasets
Noviee wewvielloe
Relation Attribune
Designed froe
frvesHapeer Konstan
2015
Enticy Amrihuee Prediction
Augmented from
Attribuee
Augmented
Eatiy Atribune
Designed Aoedier
Outiraan
Relation Attribute
Augieetted fromr
Want 2r
27746
Di Dignetica
2
V
Relation Altribuse, ER Prediction
Augmented from Augmented fiver
Willi 00 m0
Ret Rocket
Relation Attritune
Winr
37740
Stackexchange
Emily Attribute
Augmenied from
4 METHOD
This section introduces automatic graph construction solution tackle the four challenges in Section 3.1 As discussed in Section [2.2] we consider graph construction as transformation from the original table schema with implicit relations to the final graph schema with explicit relations. We adopt an LLM as the decision maker to generate transformations automatically.
4.1 AUTOG: AN LLM-BASED GRAPH CONSTRUCTION FRAMEWORK
In previous work (Fey et al. 2024 Wang al. 2024c), human data scientists often play the gener- ator, which generates outputs based on their expert knowledge. Like humans, LLMs also demon- strate the capabilities generate molecular structures code-formatted augmentations based on prior knowledge (Wang et al., 2024a Hollmann et al.) 2023). Consequently, we adopt an LLM as generator and provide with input tabular data to generate transformations.
Following the "pre-processing, modeling, and post post-processing" pipeline of common data science engineering (Biswas al., 2022), we propose automatic graph construction framework consisting of three modules: (1) Input context module, which provides essential metadata for the LLM to make decisions; (2) Generator module, which generates the augmented data together with updated data tables based on the provided context; (3) Discriminator module, which evaluates the generated schema and output feedback. The whole pipeline demonstrated in Figure 2
Input context module. The input context mod- ule designed equip AutoG with essential insights about the input data. Given the as- sumption that AutoG processes typeless tabu- lar data, this module first performs a prelimi- nary analysis by extracting key statistics such as the number of samples, unique samples, and mode values. These statistics enable LLMs to distinguish between categorical and numerical columns, critical for downstream edge relation- ship discovery. Accurately inferring column data types can significantly influence down- stream task performance. For instance, text- valued column might be interpreted as either text (encoded as dense text embeddings) or cat- egorical (represented via learnable embeddings or transformed into self-induced relationships). The prompt for column type inference is de- tailed in Appendix D.I.I The input context module compiles the following information for LLMs: 1. Input table schema with inferred column types, 2. Statistical summary of the data (e.g., sample count, uniqueness, mode, and randomly sampled values), 3. Task description for the downstream objective, 4. Chain-of-thouse examples (further discussed in Section 4.2 The statistical summary
Prompt Instruction
Inpet
PK Valne
A
Value
generators
Value
Table
Vilue
Houristess
Vilue
Value
Ornele
Flestble
discrtminater
Generated table
Designs
Figure 2: An illustration of our proposed AutoG framework.
5

---

Published as a conference paper at ICLR 2025
plays pivotal role in guiding LLM decisions. Following Hollmann et al. 20233) include: num- ber of samples, number of unique samples, mode values, and randomly sampled column values. For example, column where unique samples equal the total sample count is likely a primary key. LLMs generate concise column descriptions based on column names and these statistics. Additionally, we incorporate similarity scores from Dong et al. (2023) as prior knowledge, allowing L.MSs to inte- grate statistics, textual descriptions, and similarity values holistically. As demonstrated in Section 5 this approach outperforms relying solely on DeepJoin's 's similarity scores (Dong et al. 2023).
LLM as generators. Based on input modules, we further leverage LLMs to generate a transformed schema. straightforward approach is to let the LLM directly generate structured outputs such as YAML (Ben-Biise et al. (2009) formatte code. However, we find that open-ended generation usually produces invalid graph structures. To address this, inspired by the idea of function calling (Schick) et al. 2024), we design basic augmentation actions based on 5 challenges of graph construction and then guide the output through chain-of- -fg-em--atummttiiinnnnnn prompts, which elaborated in Section [4.
Oracle as discriminators. After generating the graph schema, we adopt an oracle model to evaluate its effectiveness. First, we need a heuristic algorithm to convert tables into graphs. This paper con- siders two algorithms: Row2Node and Row2Node/Edge (Wang et al. 2024c). The former converts tables with at least two columns as FK and no PK into edges of a heterogeneous graph, while the latter converts other tables into nodes. We then train GML model based this graph and make final predictions. After generating the graph, we design an oracle as a discriminator to generate feedback. Such feedback be qualitative, like whether graph construction fails due grammar errors, or quantitative, like the performance of trained GML models. We discuss the oracle design in Section 4.3
4.2 GUIDED GENERATION WITH CHAIN-OF-AUGMENT
The most straightforward way to let LLMs generate schema is directly generating the YAML- formatted structured outputs. However, such open-ended generation suffers from the following pitfalls: 1. LLMs generate schema and augmentation code with grammar errors, which makes the pipeline fail to proceed automatically. 2. LLMs tend to miss those node types and relations that require multi-step augmentation. Taking the Digineti ca dataset as an example, relations may be found by first transforming set-attributed columns into proper augmented columns and then identifying the non PK-FK relations from the augmented columns. Simply generating the schema in single-step manner fails to extract such relations. To alleviate these problems, we propose guided generation with chain of augmentation. First, based on four challenges proposed in Section 3.1 we identify the following basic actions for augmentation.
CONNECTITWE TWO TwL_TTTT_LLLi. COLUMNS:: Building a PK-FK or FK-FK relationship between two columns. This action is designed to tackle challenge 1. Compared to joinable table discovery (JTD) (Dong et al.) 2023 Hulsebos et al. 2019). LLMs can automatically identify the relationships without relying on humanly defined thresholds. Moreover, LLMs can also generate better join discovery results by utilizing meta information such as column statistics and descriptions. We validate this in Section
2. GENERATE_NEN, TABLE: Inducing a new table from the original table via moving columns with- out changing any values. This can be viewed identifying multiple node or relation types from the original table. This action is designed to tackle challenge 2. Moreover, this action can also be used to normalize the original data satisfy certain normal forms (Ghosh et al., 2018)). which enables our system to be self-contained without relying on external normalization tools.
REMOVE (ADD) PRIMARY KEY: Combined with proper heuristic methods, this action can change the type of table (as a node or an edge type) in the generated graph. This action is designed to tackle challenge 3.
4. UNFOLD WDULTICCATEGORY_CCOO
This action is usually provided by data normalization tools. We include it to make our system self-contained.. Its difference compared to traditional data normalization tools is that 1. LLMs should determine whether unfolding such columns is helpful; 2. LLMS need to determine the data types of the unfolded columns.
Then, LLMs will determine which actions to take by looking at input context introduced in Sec- tion 4.1 and chain of thought demonstrations. For each of these actions, we provide a of thought demonstration showcase its usage. Specifically, we find that chain-of-thought (CoT)

---

Published as a conference paper at ICLR 2025
prompts (Wei al. 2022) are critical to action generation. As motivating example, LLMs tend to merely find those columns with identical names to build non- PK-K relationships without CoT. Only after introducing CoT demonstrations general non-PK-FK relationships with different column names. After generating the draft action based on provided information, we empirically find that adding a self-reflection step with prompt like "Please double check and fix any errors" can further improve LLMs' capability to generate proper actions. The complete prompt design can be found in Appendix D.1 To determine the termi- nation step, we add null action to the action space and set a hard threshold T to limit the maximum number of actions, typically set to 10 for our proposed datasets.
LLMs utilize the statistics of columns to find more
4.3 DESIGNING ORACLE TO GENERATE FEEDBACK
After generating the schema candidates, need an oracle to evaluate their effectiveness and thus choose the best schema. Despite LLM's capability to generate schemas based on prior knowledge, they cannot quantitatively predict how different schemas affect downstream task performance (Wang) eet al. 2024b). We thus need a graph-centrie model to generate the feedback.
The main challenge of designing quanti- tative oracle is to 1. efficiently obtain the performance estimation, 2. the performance measure should present a mhodel-agnostic es- timation of the graph schema's 's influence on downstream tasks. We first explore the pos- sibility to speedup the estimation process: (1) Condensating the graph (Hashemi et al.] 2024). improving the evaluation efficiency by training and testing on smaller graph; (2) Adopting an early-stage training metric, such as the validation set performance. We
Table 2: Evaluating different oracles by quality and efficiency. For sampling, we set the ratio to 30%. For carly-stage validation performance, we set to 10% of to- tal epochs (should be according different datasets). The pre-processing set as the basic unit; all other time rounded to an integer.
Discrepancy
Training (mode)
Training (link)
Process
Full
29s
300x
lx
Sumpled Actively sampled
0.75
16x
95x
1x
0.75
16x
95x
3x
Early metric
10x
52x
then compare these methods in terms of their effectiveness and efficiency. Specifically, we randomly sample three groups of schemas (in total 36, with distinguishable performance) from the proposed datasets. Then, we let different oracles generate orders for each group and measure the normalized Kendall's tau distance (Kumar & Vassilvitskii 2010) ones generated by regular GML models. From the experimental results in Table 2 we find that only the carly-stage validation performance can reasonably estimate the downstream task performance. In AutoG, we use this strategy to speed up the estimation of some large-scale datasets. We then discuss developing model-agnostic per- formance estimation. We find that sometimes there presents a gap between different GNN models such as RGCN (Schlichtkrull et al.) 2018) and heterogeneous graph transformer (Hu et al. 2020b). To address this problem, we adopt a simple strategy to use the average performance adopted from basket of GNN models as the final oracle score. In this paper, consider RGCN (Schlichtkrull cct al., 2018), RGAT (Veličković et al. 2018 [Wang et al.) 2024c). HGT (Hu et al.) 2020b). and PNA (Corso et al. 2020) as the basket of GNN models, which are four widely adopted GNN models for heterogeneous graphs (Wang et al. 2024c).
4.4 GENERATING DIVERSE CANDIDATES
Despite AutoG's capability to generate graph schemas automatically, its decision relies on LLMs' reasoning and may not always generate optimal results (as shown in Section One remedy for this issue is to LLM generate a set of candidate results and then select the best based on performance measured by oracles. When AutoG is adopted as tool for data scientists, diverse candidates can be proteproe to help them make decisions. To produce diverse schemas, we run the algorithm multiple times and choose the candidates with the best oracle score the final selection. We denote such version of AutoG as AutoG-A, while the alternative directly uses the final state as AutoG-S. We demonstrate when AutoG-S is not good enough to produce the best graph schemas and what's the underlying reason in Section
7

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTAL RESULTS
In this section, we systematically evaluate the AutoG framework on the proposed benchmarks from the following perspectives: 1. Quantitative evaluation of the performance of different graph con- struction methods. 2. In-depth analysis of the working mechanism of AutoG and its limitations.
5.1 EXPERIMENTAL SETTINGS
To investigate the impact of different graph construction methods, fix the GML model to check the downstream task performance according to different graph schemas. Specifically, we commonly used baselines heterogeneous graphs, including RGCN, RGAT, HGT, and RPNA (Wang al.,
2024c). This section presents RGCN's performance and shows others Appendix [._1 We select Claude's Sonnet-3.5 as the backhone of L.LMs and discuss the influence of different L.MSs in Ap-
pendix E.2 We consider the following baseline methods: (1) XGBoost (Chen & Guestrin, 2016) and
DeepFM (Guo al., 2017). 7. (2) TabGNN (Guo al., 202T). (3) JTD with Row2Node/Edge (Dong et al. 2023] Gan et al. 2024). (4) Graph schema designed by human experts. We detail the expert
schema design in Appendix E.3 (5) Vanilla graph schema: Based on the expert schema, we remove relations introduced by experts to construct a minimal relation subset supporting graph construction, and then generate relations with heuristics. should noted that JTD and TabGNN take an easier setting because they augmented from the vanilla graph schema rather than the original schema without any information; otherwise, they can not even generate valid graphs.
5.2 QUANTITATIVE EVALUATION
Table shows the performance of different graph construction methods. Models' performance is used to determine the quality of graphs. We consider both the individual model performance and the average model set's performance. We show the performance of RGCN, ranking of RGCN, and ranking of average performance in Table The metrics for each task are shown in the second col- umn, and the ranking calculated based on the average ranking of each task. To prevent shortcuts by looking at column names, we change all identical column names to different names.
Table 3: Evaluation of different graph construction methods on proposed datasets. The best is in bold, second best is underlined, and third best is double-underlined.
Dataset
Task
XCBOOST
DeepFM
TabGNN
Vanilla Schema
JTD Schema
AutoG
Expert
N/A
N/A
TabGNN
R2N
R2NE
R2N
R2NE
AutoG
Expert
Datasets with Single Downstream Task
IEEB-CIS
Fraud (AUC)
90.14
90.28
75.38
89.17
89.17
89.17
89.17
89.17
87.28
RetailRocket
(AUC)
50.35
49.33
82.84
50.45
49.90
50.82
74.78
84.70
Movielens
Ratings (AUC)
53.62
50.93
55.34
57.34
61.20
54.55
57.23
69.10
69.73
Outbrain
CTR (AUC)
50.05
$1.09
62.12
49.33
52.06
49.35
52.23
61.32
62.71
AVS
Repeat (AUC)
52.71
$2.88
$4.48
47.75
48. 84
53.27
53.27
56.03
55.08
Datasets with Multiple Downstream Tasks
Venue (Acc)
21.95
28.19
42.84
27.24
46.26
21.26
46.97
49.88
49.66
MAG
Citation (MRR)
3.29
45.06
70.65
65.29
65.29
72.53
81.50
80.84
80.86
(Ace)
28.09
52.77
30.90
53.07
53.07
54.09
35.35
Diginetica
CTR (AUC)
53,50
50.57
50.00
68.44
65.92
50.05
50.00
75.07
75.07
Purchase (MRR)
3.16
5.02
5.01
5.64
7.70
11.37
15.47
36.91
36.91
Churn (AUC)
58,20
$9.84
78.27
77.67
85,58
84.85
86.92
Stackexchange
Upvote (AUC)
86.69
87.64
85.28
86.45
86.47
88.61
67.98
87.43
88.61
Ranking (RGCN)
5.9
5.2
4.7
2.0
2.0
Ranking (Average)
5.8
5.3
4.5
4.4
3,3
2.0
2.0
From the experimental results, make the following observations
AutoG generates high-quality graphs: The AutoG method we propose can surpass other auto- matic graph construction methods and reach close to the level of human experts.
methods: Heuristic-basee methods can only be applied to some special cases. We demonstrate its superiority against JTD with an example of MAG. When applying JTD to the MAG dataset. JTD ranks the column "pa- per cite" from Table "Cites" and column "paper writer" from Table "Writes" the second most
AutoG's superiority against heuristic-basee
automatic discovery
8

---

Published as a conference paper at ICLR 2025
similar pairs, which is incorrect. AutoG avoids this problem by making decisions based meta- data provided in context.
The same graph may not be adequate for different downstream tasks. On the MAG dataset, we observe that the expert-designed graph is not optimal for the year prediction task and is much worse than the original schema. We diseuss this problem in more detail in Section 5.3.
5.3 IN-DEPTH ANALYSIS
To better understand the effectiveness of AutoG, we further study the effect of its components. We conduct three experiments: (1) Comparing AutoG variants, and studying when AutoG or expert schema fails deliver promising downstream task performance. (2) Studying the necessity of each prompt component and AutoGi's performance on synthetic data with anonymous columns.
5.3. STUDYING AUTOG VARIANTS
We consider variants of AutoG: AutoG-S and AutoG-A, where AutoG-S directly adopts the final output state while AutoG-A uses an oracle to select the state. We also consider AutoGi O, which conducts open-ended generation. As shown in Table 4 we draw the following con- clusions: 1. Close-ended generation is neces- sary for valid schema generation. 2. Oracle is often unnecessary, meaning LLMs can gener- ate good candidates merely based on prior knowledge. However, two bad cases exist: (1) Expert schema doesn't work well: An example is the year prediction task on the MAG dataset. Taking a deeper look at the generated graph statistics, we find that when predicting the venue of "Paper", the adjusted homophily (Lim et al., 202T) of labels based on metapath 'Paper-Author-2 Paperr" is 0.156.
Table 4: Ablation studies for closed-ended generation and oracles
Dataset
Task
Valid
Performance
A
Amoti
Auto-A
AutoGi-S
Auto-A
Vanue
54.09
IEBE-CTS
Pruad
89.17
RetailRockets
CVR
74.78
While for year prediction, the adjusted homophily is only 0.02. This can be viewed as exten- sion of the heterophily problem (Lim et al., 2021) towards the RDB data, and an effective graph construction algorithm should address this problem by eliminating harmful relations. Although we deliberately introduce chain-of-thought prompts in the graph construction. AutoG still needs to rely on graph oracle deal with this problem. (2) All graph construction methods don't work well: An example is the IEEE-CIS dataset. We find that despite the reasonable data normalization process (for example, generating a new table based on columns forming an independent entity "MatchSta- tus", full example can be shown in Appendix F..3). such graph construction negatively affects the performance of GML. model compared to the original one. This phenomenon corresponds to a more challenging scenario in which we must infer beneficial network effect (Lee et al. (2024). Compared to the homophily/heterophily problem, which may be mitigated by calculating graph statistics, infer- ring network effects is much more challenging, especially when the semantics of columns can't 't tell network effects. Generally, determining whether a graph is beneficial and telling whether a graph can be constructed are key challenges when applying GML to industrial data.
5.3.2 WORKING MECHANISM OF AUTOG
Table 5: Ablation studies of different AutoG prompt components. "Orig" stands for the original schema with original names. "Anon" stands for the anonymous column names. "3/3" means 3 of the 3 expected actions have all been generated.
Despite the promising performance of Au- toG, LLM as generators is composed of complicated prompt designs, which makes it challenging to understand the role of each component and how they may be applied to more general types of tabular data (for ex- ample, ones with anonymous columns). We thus further study the influence of different prompt components. In our prompt design, we have considered the following compo- nents: 1. the semantic information of the column (column name); the statistical meta- inntoraifoottooon of the column; 3. the examples given in
Challenge
Challenge 2
Challenge 3
Orig
Anon
Orig
Anon
Orig
Anon
Default
3/3
1/3
2/3
1/3
0/2
No COT
1/3
0/3
1/3
0/3
0/2
No stats
1/3
0/3
1/3
0/3
0/2
1/2
No demo
0/3
0/3
0/3
0/3
0/2
0/2
the prompt; 4. the chain of thought demonstrations for each action. Specifically, we built synthetic dataset based on MAG to include the challenges 1 - 4 proposed in Section 3.1 and ensure the test
data is not included in the pre-training set of L.MMs. Compared quantitative evaluation, we di-

---

Published as a conference paper at ICLR 2025
rectly study whether LLMs can generate the required actions for better graphs. As shown in Table we observe the following conclusions: 1. Demonstration is necessary for AutoG to generate valid actions. 2. Both COT and statistics are critical to the graph schema generation. Specifically, we find that LLMs will only find trivial augmentations (for example, non-PK-FK relations with identical column names), which means COT is the key for L.MSs to conduct deep reasoning and to utilize the statistics sufficiently. 3. Semantic information of the column names is vital for the performance of AutoG, which a limitation of AutoG.
RELATED WORKS
Recently, GML has been widely adopted to capture the structural relationship across tabular data al.] (224). One of the key challenges lies identifying graph structures from tabular data that can benefit the downstream tasks. Early endeavors in database management mine relationships across databases using rule-based methods (Yao & Hamilton, 2008 [iu et al. 2012 Abedjan al,) 2013 Koutras et al., 2021). One limitation of these methods lies in their scalability towards large- scale tables. The rise machine learning has led to two ML-based approaches: heuristic-based and learning-based methods. Heuristic-bused methods transform tabular data into graphs based on specific rules. For instance, Guo et al. (2021) generates edge relationships based on columns with categorical values in the table, resulting
multiplex graph through multiple columns. [Wu et al. (202T) and You et al. (2020) create a bipartite graph based on each row representing a sample and each column representing feature, where You al. (2020) further supports numerical values by storing them as edge attributes. [u et al.
(2022) generates a hypergraph by treating each row as hyperedge. A major challenge for these heuristic methods is the inability to handle multi-table scenarios effectively. Row2Node (Fey et al. 2024) and Row2Node/Edge (Wang et al. 2024c) are proposed for multiple tables with explicit key relationships. Bai et al. (202)) designs an end-to-end model for RDB prediction tasks. These methods are still limited to tables satisfying RDB specifica- tions. Learning-basse methods aim to learn edge relationships automatically based on the correlation between features. [hen et al. (2020) and Franceschi et al. (2019) leverage graph structure learning to learn the induced edge relationships between each sample. However, learning- -aand methods suffer from efficiency issues, and their effectiveness is challenged by Errica (2024) when adequate supervision is provided. Koutras et al. (2020) leverages knowledge graph to build relation graph across different columns and extract potential structural relationships. Dong et al. (2023) leverages a language model embedding to detect similar columns in the table and thus extract those related columns. To study the effectiveness of different GML methods for tabular data, multiple bench- marks have been developed (Wang et al., [2024c Fey et al. [2024] Bazhenov et al., (202)). However, their scopes are limited to either model evaluation (Wang et al. 2024c] [Fey al., (2024) or feature evaluation (Bazhenov et al. 2024), which leaves graph construction evaluation an undercxplored
area.
7 CONCLUSION
In this work, we formalize the graph construction problem through benchmark and propose AutoG, an LLM-LaMaa -.sdd- solution for automated graph generation. Our extensive experiments demonstrate that graph construction critically impacts downstream task performance. However, automatic graph con- struction remains highly challenging; AutoG serves as preliminary step, currently addressing only moderately complex scenarios. Looking ahead, we identify three fundamental challenges pivotal to this field: (1) establishing criteria to evaluate whether a graph structure offers measurable advan- tages over non-graph methods; (2) determining the fcasibility of deriving a beneficial graph from multi-tabular datasets; and (3) isolating core relational patterns essential for task performance while pruning superfluous connections. Resolving these challenges imperative to bridge the gap be- tween theoretical GML advancements and their robust, scalable application in real-world industrial settings.
8 ACKNOWLEDGMENTS
We thank the authors of 4DBInfer (Wang et al., 2024c) for open- open-soureing their project, which has been instrumental in enabling us develop the codebase for graph construction.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ziawasch Abedjan, Lukasz Golab, and Felix Naumann. Profiling relational data: survey. The VLDB Journal, 24:557-581, 2015.
Jinze Bai, Jialin Wang, Zhao Li, Donghui Ding, Ji Zhang, and Jun Gao. Atj-net: Auto-table-join network for automatic learning on relational databases. In Proceedings of the Web Conference 2021, WWW 221, pp. 1540-1551, New York, NY, USA, 2021. Association for Computing Ma- chinery. ISBN 978 4701450333 doi: 10.1145/3442381.3442 34499880.
URL https://dc...r.
1.0.14453442333 34412998888448808 3449980
Gleb Bazhenov, Oleg Platonov, and Liudmila Prokhorenkova. Tabgraphs: new benchmark and insights for learning on graphs with tabular features, 2024. URL
het/ wtw.ooorwwmmm00000000000
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-sequence learning using gated graph neural networks. arXiv preprint arXiv: 1xXiv:186.0096 180609983, 2018.
Oren Ben-Kiki, Clark Evans, and Brian Ingerson. Yaml ain't markup language (yaml)(tm) version 1.2. YAML. org, Tech. Rep, 359, 2009.
Sumon Biswas, Mohammad Wardat, and Hridesh Rajan. The art and practice of data science pipelines: A comprehensive study of data science pipelines in theory, in-the-small, and in-the- large. In Proceedings of the 44th International Conference on Software Engineering, pp. 2091- 2103, 2022.
Peter Pin-Shan Chen. The entity-relationship towel-toward a unified view of data. ACM Trans. Database Syst., 1(1):9-36, 36, mar 1976. ISSN 0362-5915. doi: 00.14455220434322444... URL
0ttp://0..115/322433303 http:s//oaa..rr//...113333333322
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural net- works: Better and robust node embeddings. Advances in neural information processing systems, 33:19314-19321 2020.
Zui Chen, Lei Cao, Sam Madden, Ju Fan, Nan Tang, Zihui Gu, Zeyuan Shang, Chunwei Liu, Michael Cafarella, and Tim Kraska. Seed: Simple, efficient, and effective data management via large language models. arXiv preprint arXiv:2310.007490 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference on Machine Learning. 2024.
Edgar F Codd. Relational database: A practical foundation for productivity. In ACM Turing award lectures, pp. 1981. Association for Computing Machinery, 2007.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. Advances in neural information processing systems, 13:13260-132711 2020.
Milan Cvitkovic. Supervised learning on relational databases with graph neural networks. arXiv preprint arXiv:2002.002046 2020.
Yuyang Dong. Chuan Xiao, Takuma Nozawa, Masafumi Enomoto, and Masafumi Oyamada. Deep- join: Joinable table discovery with pre-trained language models. Proc. VLDB Endow., 16(10): 2458 2470, June 2023. ISSN 8097. doi: 111147778/603388133033558 URL https:
7/doi. org 11t.//10..17777333 3603587

---

Published as a conference paper at ICLR 2025
Kounianhua Du, Weinan Zhang, Ruiwen Zhou, Yangkun Wang, Xilong Zhao, Jiarui Jin, Quan Gan, Zheng Zhang, and David Wipf. Learning enhanced representation for tabular data via neigh- borhood propagation. Advances in Neural Information Processing Systems, 35:16373-16384, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Federico Errica. On class distributions induced by nearest neighbor graphs for node classification of tabular data. Advances in Neural Information Processing Systems, 36, 2024.
Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. Position: Relational deep learning graph representation learning on relational databases. In Forty-first International Conference on Machine Learning, 2024. URL
Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In International conference on machine learning, pp. 1972-1982.
PMLR, 2019.
Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. Proceedings of the web conference 2020, pp- 2331-2341, 2020.
Quan Gan, Minjie Wang, David Wipf, and Christos Faloutsos. Graph machine learning meets multi-table relational data. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6502 6512, 2024.
Aindrila Ghosh, Mona Nashaat, James Miller, Shaikh Quader, and Chad Marston. A comprehensive review of tools for exploratory analysis of tabular industrial datasets. Visual Informatics, 2(4): 235-253, 2018.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: factorization- machine based neural network for ctr prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI'17, pp. 1725 1731. AAAI Press, 2017. ISBN 9780999241103.
Xiawei Guo, Yuhan Quan, Huan Zhao, Quanming Yao, Yong Li, and Weiwei Tu. Tabgnn: Multiplex graph neural network for tabular data prediction. arXiv preprint MtXiv:2108.09127 2021.
F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact, Intell. Syst., 5(4), dec 2015. ISSN 2160-6455. doi: 10.1145/2827872. URL
hittps://do..or///
Mohammad Hashemi, Shengbo Gong, Juntong Ni, Wenqi Fan, Aditya Prakash, and Wei Jin. A comprehensive survey graph reduction: Sparsification, coarsening, and condensation. arXiv preprint arXiv:2402.03355, 2024.
Md Mahadi Hassan, Alex Knipper, and Shubhra Kanti Karmaker Santu. Chatgpt your personal data scientist. arXiv preprint (t.Xiiv233051136551 2023.
Noah Hollmann, Samuel Müller, and Frank Hutter. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. A. Oh, T. Naumann, A. Globerson. K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36. pp. 44753 Curran Associates, Inc.,
2023. URL httpps:/proceed ngs nwwries.....ee cc httpp//pprrr.lllss/////////// e Bccddfc243dcccccc 7 64ebb 9e 9d0 - www.aaaaeaaaaaaaaaeeeeeeeeeeeeee waw.erenceeccceeecccccccccc
Sirui Hong, Yizhang Lin, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Lingyao Zhang, Mingchen Zhuge, et al. Data interpreter: An Ilm agent for data science. arXiv preprint arXiv:2402. 18679, 2024.
12

---

Published as a conference paper at ICLR 2025
Addison Howard, Bernadette Mouchon-Meuniee IEEE CIS, John Lei. Lynn Vesta, Marcus2010, and Hussein Abbass. IEEE-CIS fraud detection, 2019. URL httpp::: /tt:://kagglerr com/
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 11122218-22111 22133,,, 2020a.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of the web conference 2020, pp. 2704-2710, 2020b.
Madelon Hulsebos, Kevin Hu, Michiel Bakker, Emanuel Zgraggen, Arvind Satyanarayan, Tim Kraska, Çagatay Demiralp, and César Hidalgo. Sherlock: A deep learning approach to seman- tic data type detection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1500- 1508, 2019.
Arpandeep Khatua, Vikram Sharma Mailthody, Bhagyashree Taleka, Tengfei Ma, Xiang Song, and Wen-mei Hwu. Igb: Addressing the gaps in labeling, features, heterogeneity, and size of public graph datasets for deep learning research. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4284 4295, 2023.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https: //openreview. m
Christos Koutras, Marios Fragkoulis, Asterios Katsifodimos, and Christoph Lofi. Rema: Graph cmbeddings-based relational schema matching. In EDBT/ICDT Workshops, 2020.
Christos Koutras, George Siachamis, Andra Ionescu, Kyriakos Psarakis, Jerry Brons, Marios Fragk- oulis, Christoph Lofi, Angela Bonifati, and Asterios Katsifodimos. Valentine: Evaluating match- ing techniques for dataset discovery. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. (68. IEEE, 2021.
Ravi Kumar and Sergei Vassilvitskii. Generalized distances between rankings. Proceedings of the 19th international conference World wide web, pp. 571-580, 2010.
Meng-Chieh Lee, Shubhranshu Shekhar, Jaemin Yoo, and Christos Faloutsos. Neteffect: Discov- ery and exploitation of generalized network effects. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 299-312. Springer, 2024.
Cheng-Te Te Li, Yu-Che Tsai, Chih-Yar Chen, and Jay Chiehen Liao. Graph neural networks for tabular data learning: A survey with taxonomy and directions. arXiv preprint 023414, 2024.
Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. Advances Neural Information Processing Systems, 34:20887-20902, 2021.
Jixue Liu, Jiuyong Li, Chengfei Liu, and Yongfeng Chen. Discover dependencies from data review. IEEE Transactions on Knowledge and Data Engineering, 24(2):251-244, 2012. doi: 10.1109/KKD.22000.997
Yao Ma and Jiliang Tang. Deep learning on graphs. Cambridge University Press, 2021.
Emanuele Rossi, Bertrand Charpentier, Francesco Di Giovanni, Fabrizio Frasca, Stephan Günnemann, and Michael M Bronstein. Edge directionality improves learning on heterophilic graphs. Learning on Graphs Conference, pp. 25-1. PMLR, 2024.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves tools. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling, Modeling relational data with graph convolutional networks. The semantic web: 15th international conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, proceedings 15, pp. 593-607. Springer, 2018.
José Suárez-Vared Paul Almasan, Miquel Ferriol-Galmés, Krzysztof Rusek, Fabien Geyer, Xiangle Cheng, Xiang Shi, Shihan Xiao, Franco Scarselli, Albert Cabellos-Aparicio et al. Graph neural networks for communication networks: Context, use cases and opportunities. IEEE network, 37 (3):146-153, 2022.
Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çagatay Demiralp, Chen Chen, and Wang- Chiew Tan. Annotating columns with pre-trained language models. Proceedings of the 2022 International Conference on Management of Data, pp. 1493-1503, 2022.
Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S Yu, and Tianyi Wu. Pathsim: Meta path-hased top-k similarity search in heterogeneous information networks. Proceedings of the VLDB Endowment, 4(L1):992-1003, 2011.
Jiliang Tang, Xia Hu, and Huan Liu. Social recommendation: a review. Social Network Analysis 2013.
and Mining,
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
net http://wwwww..teeeooooe.eeeaaaaa
Haorui Wang, Marta Skreta, Cher-Tian Ser, Wenhao Gao, Lingkai Kong, Felix Streith-Kalthoff, Chenru Duan, Yuchen Zhuang, Yue Yu, Yanqiao Zhu, et al. Efficient evolutionary search over chemical space with large language models. arXiv preprint arXiv:2406. 16976, 2024a.
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov.
Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36, 2024b.
Minjie Wang, Quan Gan, David Wipf, Zheng Zhang, Christos Faloutsos, Weinan Zhang, Muhan Zhang, Zhenkun Cai, Jiahang Li, Zunyao Mao, Yakun Song. Jianheng Tang, Yanlin Zhang, Guang Yang. Chuan Lei, Xiao Qin, Ning Li, Han Zhang, Yanbo Wang, and Zizhao Zhang. 4DBInfer: A 4d benchmarking toolbox for graph-centric predictive modeling RDBs. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024c.
URL URL wwt.https::// petttps://openreeee
net./ forum? d=Xxm TTTOXXTTTHTRNNNNN
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In The world wide web conference, pp. 2022-2032, 2019.
Xiao Wang, Deyu Bo, Chuan Shi, Shaohua Fan, Yanfang Ye, and Yu Philip. A survey on hetero- geneous graph embedding: methods, techniques, applications and sources. IEEE Transactions on Big Data, 9(2):415 4366 2022.
Yanbang Wang and Jon Kleinberg. On the relationship between relevance and flict in online social link recommendations. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 16708-36725. Curran Associates, Inc., 2023. URL https::/prrceeedaggg www.eeeee....................... cc cw/paper http://aaer_fileeseeeeeeee wwt.s//pppee//22//2.....
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thougee prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-248311 2022.
Felix Wong, Erica Zheng, Jacqueline A. Valeri, Nina M. Donghia, Melis N. Anahtar, Sato- taka Omori, Alicia Li, Andres Cubillos-Ruiz, Aarti Krishnan, Wengong Jin, Abigail L. Man- son, Jens Friedrichs, Ralf Helbig, Behnoush Hajian, Dawid K. Fiejtek, Florence F. Wagner, Holly H. Soutter, Ashlee M. Earl, Jonathan M Stokes, L.D. Renner, and James J. Collins. Dis- covery of a structural class of antibiotics with explainable deep learning. Nature, 2023. URL https: //api semanteenn nt comanticschon
14

---

Published as a conference paper at ICLR 2025
Qitian Wu, Chenxiao Yang, and Junchi Yan. Towards open-world feature extrap- olation: An inductive graph learning approach. In M. Ranzato, A. Beygelz- imer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19435-19447. Curran Associates, Inc.,
2021. URL httpss://proceed ngs neurips wtt.iisss.ccc.peee wwtppppp.ppp..eeeeeeee FJle/ALLCa2T777735553336662222 220 72 Paperrapppppra.
Carl Yang. Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. Heterogeneous network repre- sentation learning: A unified framework with survey and benchmark. IEEE Transactions on Knowledge and Data Engineering, 34(10):48544477 2020.
Hong Yao and Howard J. Hamilton. Mining functional dependencies from data. Data Min. Knowl. Discov,, 16(2)197-219, April 2008. ISSN 1384-5810.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp. 974 983, 2018.
Jiaxuan You, Xiaobai Ma, Yi Ding, Mykel J Kochenderfer, and Jure Leskovec. Handling missing data with graph representation learning. Advances in Neural Information Processing Systems, 33: 19075-19087 2020.
Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. Data-copilot: Bridging billions of data and humans with autonomous workflow. arXiv preprint arXiv:2306.07209 2023.
Yuge Zhang, Qiyang Jiang, Xingyu Han, Nan Chen, Yuqing Yang, and Kan Ren. Benchmarking data science agents. arXiv preprint arXiv:2402.17168, 2024.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. The Twelfth International Conference on Learning Representations, 2024. URL URtphhtss:///oeeeeeeee Id==on9c6yy.in
net ffrrum?
Marinka Zitnik, Monica Agrawal, and Jure Leskovec. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13)i457-i466, 2018.
A MORE PRELIMINARIES
A.1 DATA TYPES
In this paper, consider the following data types fcategory, numeric, text, primary key(PK), foreign key (FK), set, timestamp).
category: A data type representing categorical values. For example, column with three possible values ("Book", "Pen", "Paper"" is of the category data type. numeric: A data type representing numerical values. This can include integers, floating-poii numbers, or decimals. For instance, a column storing ages or prices would typically be of the numeric data type.
text: A data type representing textual data. This can include strings of characters, sentences, even paragraphs. A column storing product descriptions or customer reviews would be of the text data type.
primary ry key (Px): special type of column or a combination of columns that uniquely iden- tifies each row in table. It ensures data integrity and is often used to establish relationships between tables.
combination of columns in one table that refers to the primary in another table. It creates a link between the two tables, enabling data rela- tionships and maintaining consistency. set: A data type representing a collection of values. It is often used to store multiple choices or options associated with particular record.
foreign.key (FK): A column
15

---

Published as a conference paper at ICLR 2025
t timestamp: A data type representing time. It's used define the time-based neighbor sampler and prevents data leakage.
A.2 EXAMPLES OF DATA FORMATS
We follow Wang al. 2024c to represent the table schema a YAML-formatted configuration file. An example is shown below. An example original schema plot shown in Figure 3 The original schema only presents limited relations, which may result in ineffective graph for downstream tasks. Ssowss an example of augmented relations schemas. With augmented tables including Company, Brand, Category, Customer, and Chain, the resulting graphs will benefit downstream tasks.
tables:
2 name: History
3 source: data/history pqt
4
format: parquet
5
columns:
name: chain
7
dtype: category
8
name: market
9
dtype: category
name: offerdate
dtype: datetime
12
name: id
13
dtype: primary key
14
repeater
dtype: category
name: offer
17
dtype: foreign_key
18
link_to: offer
time column: offerdate
Transaciion
Brand VARCHA
VARCHAR
History
Diler
-ompany VARCHAR
DATETINE
WARCHAS
VARCHAL
dest VARCHAR
GIAT
company
productmessure WARCHAR
Id: CHARG32!
offercalue ABBAY
offer: CHARL32!
offer (CWAR33))
purchaseamount ARNAY
purchaseauaniity
id (CHMA)37)
Figure 3: The original schema for the dataset AVS
B DATASETS
Movielens is a collection of movie ratings and tag applications from MovieLens users. This dataset is widely used for collaborative filtering and recommender system development. We adopt the tabular version from the original website. Expert schema is designed by ourselves.
MAG heterogeneous graph dataset containing information about authors, papers, institutions, and fields of study. We adopt the tabular version from Wang et al. (202cc and generate the original version by removing relations added by experts.
AVS (Acquire Valued Shoppers) is a Kaggle dataset predicting whether a user will repurchase a product based history sessions.
16

---

Published as a conference paper at ICLR 2025
Offer
category CHAR(32]
company CHARL32]
company
Company
offervalue AMMY
id CHAR(32)
quantity: ARPAY
offer CHAMU32
categore
Brand
id CHAI(32)
Vansaction
compary
brand: CHAR(33)
category: : CHAR(32)
brand
chain
Category
company CHARL32)
CHARC32)
category
productmessure VARCHAR
productsize ARRAY
Customer
offe
purchasequatiity ARRAY id CHAR132)
chain
History
Chain CHARL37)
offerdate: DATETIME
Id CHAR(32)
1d (HAR(32)
offer:
Figure 4: The new schema for dataset AVS with augmented relations
IEEE-CIS is a Kaggle dataset predicting whether a transaction is fraudulent. We adopt the original version from the website. Expert schema is designed by ourselves.
Outbrain is Kaggle dataset predicting which pieces of content its global base of users are likely to click on.
Diginetica ca is a Codalab dataset for recommendation system.
Retailrocket a Kaggle dataset for recommender system. stackexchange is a database from Stackexchange platform.
C MORE RELATED WORKS
LLMs for automated data science. Our work is also related to applying LLMs to automated data science. The core principle of these works lies in adopting the code generation capabilities of LLMs to automatically generate code for data curation (Chen et al. 2023), data augmentation (Hollmann al. 2023), working as general interface for diverse data manipulation (Zhang et al.) [002]] Hong et al. 2024 Hassan et al., 2023). Zhang et al. (2024) proposes benchmark to evaluate the capabilities LLMs various data science scenarios. Compared the methods adopted in these works, AutoG adopts close-ended generation via function calling to ensure the correctness of generation. Besides black-box I.LMs, (Suhara al. (2022) fine-tunes and utilizes pre-trained language models various data profiling tasks, such column annotation.
Learning on heterogeneous graphs Heterogeneous graphs featuring multiple node and edge types naturally abstract relational database data. Learning representations within these graphs often rely on meta-paths Yang et al. (2020), which transform heterogeneous relations into homogeneous sets. Early methods focused similarity measures derived from meta-paths Sun et al. (2011). With the advent of Graph Neural Networks (GNNs), approaches like HAN (Wang et al. 2019) ex- tract multiple homogeneous graphs based meta-paths for individual encoding. MAGNN (Fu
17

---

Published as a conference paper at ICLR 2025
al. 2020) further accounts for the roles of intermediate nodes in meta-paths. Alternttivly, RGCN (Schlichtkrull et al. 2018) and G2S (Beck et al. 2018) emphasize relational graphs, where edges carry rich semantic information.
D MORE DETAILS ON METHODS
D.1 PROMPT DESIGN
Our prompt design is demonstrated below. The key prompts include: 1. prompt used to infer the initial type of table columns 2. prompt used to generate augmentation actions
D.1.1 DATA TYPE INFERENCE PROMPT
Now you will be given list of tables and columns, each one with the following format:
2 Analysis for Table <name of the table>:
3 Column: <name of the column 1>
4 Max: <max value of the column>
5 Min: <min value of the column>
6 Mode: <mode value of the column>
7
Sampled Values: <list of sampled values>, for example, l'valuel', value2', 'value3']
8
Column: <name of the column 2>
9
Max: <max value of the column>
10
Min: <min value of the column>
11
Mode: <mode value of the column>
Sampled Values: <list of sampled values>, for example, ('valuel', value2', 'value3']
13
14
You should identify the data type of each column. The data types you choose from are:
16 ['float', 'categozy', 'datttmm'' 'text', 'multil coultil_category"
17 float: The column is probably float-type embedding tensor. There should be (nearly)
redundant values.
category: The column probably a categorical column.
datetime: The column is probably a datetime column. Only full datetime
values should be considered, some columns presenting only year or category.
month or day should be better considerd
20 text:
The column is probably text column. There should be lot of unique values. Otherwise it will probably a category column.
Moreover, we should expect texts with natural semantics, otherwise it 's probably category column.
21 multi__ctegooryy The column is probably multi-category column. Usually column value is list.
this means It should be noted that if the column is probably an embedding type, then directly put it to the float type.
23 Then, you should output discription of the column, for example:
24 "This column is probably representing the ID from 1 to of users in the system, as has a lot of unique values."
25 Output the results with the following format:
26
27
"<name of the table>":
28
"<name the column 1>"; ("<data type the column 1>", "<
description of the column 1>"),
29
"<name of the column 2>"; ("<data type of the column 2>", "< description of the column 2>")
30
}
32
33
34
In description, if you see two columns are very similar and may represent the same thing, you should mention it.
18

---

Published as a conference paper at ICLR 2025
D.1.2 AUGMENTATION GENERATION PROMPT
Imagine you are an expert graph data scientist, and now you are expected
to construct graph schema based on the original
2 inputs. You will be given original schema represented in the dictionary format:
3
<data>
4 1. dataset Lnamm: name of the dataset
5
2. tables: meta data for list of tables, each one will present
following attributes
1. name: table name
7
2. source: source of the data, can either be numpy npz file or
parquet file
8
3. columns: list of columns, each column will have following
attributes
9
1. name: column name
10
2. dtype: column type, can be either text, categorical, float primary\_kes key, foreign\ key, multil category. primary key and foreign key are
special types of categorical columns, which presents structural relationship with other tables. Multil category means this column is of list type, and each cell main contains a list of categorical values. After a column is set primary or foreign) key, it should not be changed to other types.
3. link|_to (optional): if this column is foreign key, point to which primary key from which table
12 3. statistics of the table: statistics of the column value of tables. These statistics can be used to help determine the characteristics of the columns. For example, if categorical column only contains one unique value,
then creating node type based on this column can result in a super node, which is not ideal for graph construction.
15 You should also determine whether two columns represent the same thing based these statistics.
4. Dummy table is special type of table. It's not explicitly defined with a table slot. It's defined in other tables, such as :("name": "nation", "dtype": "foreign\_key" CountryID")). In this case, "Country" is dummy table, which is not explicitly defined in the tables slot.
17
"linkl_to": "Country.
</data>
20
Here are the documents of the actions:
21
22 (actions)
24
25 Now, you need to
26 1. Actively think about whether any of the four actions should be
conducted; not, you can select "None" and halt the program.
2. output all actions you can think of from the above list to perform, and output your selection in the following format. It should be noted that for those actions with sequential relation like one new categorical column generated after expanding multi-category column, you don't need generate in one round.
28
29 <selection>
30 [['' expelaaaiiinn <explanation for the selection>, 'action'; <first action>, 'parameters': <parameters the first action> }),
31 ( ['exxllnation': <explanation for the selection>, 'action': <second action>, "parameters': <parameters for the second action> },,
32
33 </selection>
35
36 3. If not more action, output cellett.opxoonesssaletttioo.
37
38 Example:
19

---

Published as a conference paper at ICLR 2025
39 (example)
41 History Actions:
42 [history]Lactions]]
43
44 <input>
<dataset <_teet_ssstt>>
46 (stats)
47 s/dataset\_statee stats>
48 <task>
49 [task)
</task>
51 <schema>
52 (input schema) }
53 </schema>
Here we gives the similarity score of each column pair, you can use this
information to determine whether two columns may be joinable. The
similarity score is scaled to [o, 1], the larger means the more similar.
55 <similarity>
(jtd)
57
</similariy>>
58 </input>
59 Return your output in the json format inside <selectionx//alettiion
Specifically, there five key components for action generation prompt.
First, we give the document of each implemented action.
1 Here is the introduction of the function generate_or_comee htaprtteeo_ooonntttcuuuueeeee_e
2 Description:
3 This function be used in two ways:
4 1. Generate dummy table with only primary key
5 2. Turn an existing column with categorical type to an existing dummy
6 table
col_name" must be a column with category type
7 Parameters:
dbb: the database object
9 @ase_table_name the name of the original table
10 orig_col_name: the name of the original column in the original table,
this should be column with category type 11 table name: the new col name: the name of the new column
of the
table to be created/connectted
created/connected
14 Here is the introduction of the function connect_two_colll
15 Description:
Connect two columns,
function can be used for the following case. Always put the column with category type in table 1.
17 1. A category column in table is connected to a category column in table 2, in this
new dummy table will be created 18 2. category column in table is connected to primary key column in table 2, in this case, foreign key column. In case 2, table 2 col name must be a primary key column
column table will turned into
19 3. category column in table is connected to non-category and non- trick
primary key column table 2, this case, will called Surrogate Key.
20 4. If the column in table is already a foreign key, then in this case it's probably other fk columns totti--olumn-piin--tooooo case, we need to update
Parameters:
22 dbb: the database object
23 table 1 name: the name of the first table,
24 table 1 colll name: the name of the column in the first table, this should
column with category type
20

---

Published as a conference paper at ICLR 2025
25 table the of the second table table
name of column the second table, this should be a column with category type
27
28 Here is the introduction of the function explode multi sategory_colume
29 Description:
Explode a multi-category column into multiple columns. You should
determine whether to use this function. If you don't explode a multi- category column, it will be treated as single category column automatically.
31 Parameters:
dbb: the database object
33 original_table name of the original table where the multi-category
column is located
34 multi col: the
of the milticategory column
primary_keer column: the name the primary key column in the original table
36 new_table_name the name of the new table to be created
37 new_col_nee _ame: the name of the new column to be created
38 dtype: the data type of the
column, if set to "foreign_ken key", this table will contain only "foreign_keye keys". In this case, means you only want to use this column's s relaion. If set to other types, this table will contain the original column's values, and a primary key this column's values.
will be added, this
you want
Here is the introduction the function
41 Description:
42 Generate non-dummy table with columns in the original table
43 Parameters:
dbb: the database object
45 base_tabe name: the name of the original table
46 cols: the list of columns to be included in the new table and removed from the original table
new_table_nnt name: the name the new table to created
49 Here is the introduction of the function remove_ hamove_primary_ker
50 Description:
51 Remove primary key constraint from column in the original table If the column just an index, then the column will be removed from the table.
53 For example, if the schema is like {
54 ("name": "id", "dtype": "primary_key" ) ("name": "user", "dtype": sereign_key"s "link_to": "user. userID"), ("name": "book", "dtype": "foreign_key", "link_to"; "book. bookIDo
55
57
58 In such
it's clear that this table represents the role of edge, while the presence of primary key prevents heuristic turn this table into an edge. Primary key is not needed this case.
59 In such case, we will remove the primary key constraint from the column.
60 Parameters:
61 dbb: the database object
62
name: the name the original table
col_name: the name of the column the original table
64
65 Here is the introduction of the function add primary key:
66 Description:
Add primary key column the original table
68 Parameters:
69 dbb: the database object
71 col :ame: the name of the original table
70 base_table name:
the of the newly added primary key column
Second, for in-context learning examples, we use the following prompt.
Table:
Paper
21

---

Published as a conference paper at ICLR 2025
2
3
"Column": "PaperID",
4
"data type": "primary_kay
5
6
7
"Column": "Title",
8
"data type": "text",
9
"Number of unique values": 10000,
10
"Number of nan values"; 0, "Number of total values": 10000, "Mode values": "Transformers" "5 sampled values":
11
"Transformers"
15
"Graph Neural Networks", "Reinforcement Learning", "Meta Learning", "Computer Vision"
16
19
]
20
21
"Column": "Authors",
23
"data type": "mulliimaaatiiieeeeeeeeeeeeii coultti_ategory",
24
"Number of unique values": 987,
25
"Number of
values":
"Number of total values": 74320,
"Mode values"; "Yann LeCun",
28
"5 sampled values":
29
"Yann LeCun", "Geoffrey Hinton",
30
"Yoshua Bengio", "Fei-Fei Li", "Jitendra Malik"
32
33
34
37 "Column": "Journal",
38
"data type": "category",
39
"Number of unique values": 100,
"Number of nan values": 0,
41
"Number of total values": 10000,
42
"Mode values": "Nature",
43
"5 sampled values";
44
"Nature",
45
"Science",
46
"NeurIPs",
47
"ICML",
48
"CVPR"
1
50
51
52
"Column": "Year"
53
"data type": "float",
55
56
"Column": "Keywords",
"data type": "caategoy", "Number of unique values": 100, 59
57
58
"Number of nan values": 0, "Number of total values": 10000,
60
61
"Mode values": "Machine Learning",
62
"5 sampled values":
"Machine Learning",
64
"Deep Learning",
65
"Graph Neural Networks", "Reinforcement Learning",
66
22

---

Published as a conference paper at ICLR 2025
67
"Meta Learning"
68
]
69
70
71
"Column": "Abstract",
72
"data type": "text",
"Number of unique values": 10000,
74
"Number of nan values": 0, "Number of total values": 10000,
75
76
"Mode values": "This paper presents networks.""
model for graph neural
"5 sampled values":
78
"This paper presents a new model for graph neural networks.", "This paper introduces new reinforcement learning algorithm.", model for transformers.";
79
80
"This paper presents
"This paper presents new model for meta learning."
82
"This paper presents a new model for computer vision."
83
]
84
85
"Column": "Category",
87
"data type": "category",
88
"Number of unique values": 10,
89
"Number of
values":
"Number of total values"; 10000,
91
"Mode values": 3,
92
"5 sampled values":
93
3,
94
1,
96
6,
97
9
98
101 "Column": "ItemID",
102
"data type": "foreign_key key"
103
104
Table: Journal
105
106
"Column": "JournalID",
107
"data type": "primary.kev"
108
109
110
"Column". "Name",
111
"data type": "text",
112
"Number of unique values": 100,
113
"Number of nan values": 0,
114
"Number of total values": 100,
115
"Mode values": "Nature",
116
"5 sampled values":
117
"Nature",
118
"Science",
119
"NeurIPS",
120
"ICML",
121
"CVPR"
122
]
123
124
125
"Column": "ImpactFactor",
126
"data type": "float"
127
128
129
"Column": "Country",
130
"data type": "category",
23

---

Published as a conference paper at ICLR 2025
131
"Number of unique values": 10,
132
"Number of nan values":
133
"Number of total values": 100,
134
"Mode values": "USA",
135
"5 sampled values": [ "USA"
136
137
"USA"
138
"Canada",
139
"UK",
140
"USA"
141
]
142
143
144
"Column": "Publisher",
145
"data type": "text",
146
"Number of unique values": 9,
147
"Number of nan values": 0,
148
"Number of total values": 100,
149
"Mode values": "Springer",
150
"5 sampled values":
"Springer",
152
"Elsevier",
153
"ACM",
154
"IEEE"
155
"Nature"
156
]
157
158
159
"Column": "PublisherLocationn",
160
"data type": "category",
161
"Number of unique values": 5,
162
"Number of nan values": 0, "Number of total values"; 100,
163
164
"Mode values": "USA",
"5 sampled values": [ "USA"
166
167
"USA"
168
"Canada",
169
"UK",
170
"USA"
171
]
172
173
174 /ddtaset_stats>>
175 <tasks>
176 Now want to train model which can predict the category of paper
based the information in the paper.
177
</tasks>
178
<schema>
179
180
idaaasse_name"" "Papers",
181
"tables":
182
183
"name": "Paper",
184
"source": "data/paper.npp npz""
185
"columns":
86
("name": "PaperID", "dtype": "primary.key")
187
("name" "Title", "dtype": "text"),
188
"name" "Authors", "dtype": [multi_categorr"))
189
("name" "Journal", "dtype": "category"),
190
["name". "Year", "dtype": "float"),
["name": "Keywords", "dtype": "'aategry"); }
192
{ ("name": "Abstract", "dtype": "text"),
193
("name": "Category", "dtype": "category")
194
24

---

not similar enough to Published as a conference paper at ICLR 2025
195
196
197
"name": "Journal",
198
"source": "data/Jourral.npe""
199
"columns":
200
("name": "JournalID", "dtype": (primary_key")
("name": "Name", "dtype": "text"),
202
{ ("name", "ImpactFactor", "dtype": "float"),
203
("name": "Country", "dtype": "category"),
204
("name": "Publisher", "dtype": "text"),
205
("name": "Fublisherlocationn" "dtype": "category")
206
207
208
209
210
</schema>
211
Here we gives the similarity score of each column pair, you can use this information to determine whether two columns may be joinable. The is scaled to [0, 1], the larger
similarity
the
similar.
212 <similarity>
213 The pair with the 1st highest similarity is column "Journal" from Table
Paper" and column "Name" from Table "Journal" with similarity 0. 885
214 The pair with the 2nd highest similarity is column "Authors" from Table
Paper" and column "Name" from Table "Journal" with similarity 743
215 The pair with the 3rd highest similarity column "Authors" from Table Paper" and column "Country" from Table "Journal" with similarity
0.723
216
</ /similarity>
</ input>
218
219
220
We need to think about whether we need to one
actions:
1. First, for explode_mult multi eategory_columm the Authors of the paper are mttpeeelttttttttt_ttttoooo
in a multi-category column. Moreover, author is closely related to the category of the paper, be very useful. So, we need to explode this multi category column. 2. For connect two columns, the Journal column in the Paper table and the column Name in the Journal table are highly similar, so we can connect these two columns with foreign key constraint. Other pairs like Authors and Name, Authors and Country connected.
the relationship Authr--utt can
224 3. For generate_non cectrate_non__duue cron_dummy_taiee columns are independent columns for the entity Publisher. We can generate
the Publisher and Publisherlocation
table Publisher with these two columns.
225 4. For generate or or_coonecttttooooooooeeeeeeeeeee connect_dummyy.. columns beneficial for downstream task. We have categorical columns (Journal has been deleted in step 2, Category is the final objective) Keyword, Country, this will result in relationship Paper-Keyword- Paper and Shpe--Jooonal--oottyyoooooaaaaa target
need find those categorical
respectively. Since the
predict the category dummy table for the column Keyword since paper sharing the same keyword are highly likely to share the same category. Country may be not beneficial since it doesn't present strong semantic relationship with the category. For remove_ hammoveprrimary..e
paper, can generate
226
and add key, there's no unreasonable primary key or missing primary key in the table, so we don't t need to do this action. as a result, we have the following actions <selection>
227
228
[[[ ['l'explanation': "Author is multi-category and Paper-Author- Paper is probably useful. We set the dtype to foreign_ket key because we want to use the relation", 'action': explode multi category_columm! pparaettes': ('original table': 'Paper', 'multi cat_
col'; 'Author',
25

---

Published as a conference paper at ICLR 2025
primary key column: 'PaperID', 'new table name' 'Author', new_col_namer 'AuthorName', 'ttpe': foreign_key }
229
{( (1'explanation' 'the Journal column in the Paper table and the column Name in the Journal table are highly similar, both of them should refer to the name of the journal', 'action': connect_two_ htt.ttoww_uss',..._oo 'parameters': ('table 1 name' 'Paper', table __c____oo_______________________ pool__amme 'Journal', 'table 2 name' 'Journal', 'Name', 'new_table name': "", new_table_com col name
table_2_com. 2_col_namee :: }),
230
II'explanation': 'Publisher and Publisherlocation columns for the entity Publisher. Publisher with these two columns', 'action': generate_non, bactte__nondummyyyaee caammy_table,,. 'parameters': ('base_table_nae _ 'Paper', 'cols' ('r:::''ubbliiii ['Publisher', "Publisherlocationn] Publisher')), }
independent
generate
table
'new table name'.
{( I'eeaaaaation' 'Keyword is categorical column which can used to generate a dummy table. Country is not beneficial for the downstream task', 'action' venerate_or_conneet parameters':
htw.coonnntduuunn__eeeeeee dummy_table',
Sable_name 'Paper', 'orig_col nol__name: 'Keyword', 'new_taie table name': 'Keyword', 'new_col name' 'Keyword'
233
</selection>
The third component reflects the statistics of each column's ' data. Specifically, following Dong et al. (2023), consider the number of unique values, mode values, maximum and minimum values if numerical, and k sampled values. If this column belongs to a list type, we will also consider how many unique values will get after expanding into categorical column. We give example as follows.
Analysis for Table paper:
2
Column: paperID
3
Max: 736388
4
Min:
5
Mode:
6
Sampled Values: [311458 138871 636067 119201 468996]
7
Number of Unique Values: 736389
8
Column: label
Max:
Min: 0
11
Mode:
12
Sampled Values: [190 85 45 183 283]
Number of Unique Values: 349
Column: feat
15
Column is mutti-dimenssoonl.. Probably an embedding type. Usually not of interest
16
Column: year Max: 2019
Min: 2010
19
Mode: 2013
20
Sampled Values: [2014 2010 2019 2010 2010]
21
Number of Unique Values: 10
23
Analysis for Table Cites:
24
Column: paper_cite
25
Max: 736388
26
Min: 0
27
Mode: 732008
28
Sampled Values: [571834 223729
711055
26073 352954]
29
Number of Unique Values: 617924
30
Column: paper__ppc citer
Max: 736388
32
Min: 0
33
Mode: 428523
34
Sampled Values: [557047 417778 395162 521944
108483]
35
Number of Unique Values: 629169
26

---

Published as a conference paper at ICLR 2025
36
Analysis Table HasTopic:
38
Column: paper_name
39
Max: 736388
40
Min: 0
41
Mode: 69985
Sampled Values: [388977 701406 503766 451820 101399]
43
Number of Unique Values: 736389
44
Column: field of of study
45
Max: 59964
46
Min: 0
Mode: 14055
48
Sampled Values: [12834 15397 21310 24376 3744]
49
Number of Unique Values: 59965
50
Analysis for Table AffiliatedMith:
Column: author
53
Max: 1134648
54
Min: 0
55
Mode: 244427
Sampled Values: [377682 380472 116413 434611 284604]
57
Number of Unique Values: 852987
58
Column: institution
59
Max: 8739
Min:
61
Mode: 649
62
Sampled Values: [2315 649 4029 5285 664]
63
Number of Unique Values: 8740
64
Analysis for Table Writes:
66
Column: paper writer
paper_writer
67
Max: 1134648
68
Min: 0
Mode: 239580
Sampled Values: [613331 153535 540376 618466 462598]
71
Number of Unique Values: 1134649
72
Column: arxiv_id
73
Max: 736388
Min:
75
Mode: 522277
76
Sampled Values: [731086 691749 540097 711055 194402]
77
Number of Unique Values: 736389
The next component refers the specific task instruction given by users. an example, the instruc- tion for the "citation" task on the "Mag" dataset is "This task is to predict the venue of paper given the paper's title, abstract, authors, and publication year. You may use the meta relations between papers, authors, topics, and institutions to improve the performance.
Finally, we use the result from deepjoin as prior in the generation prompt. will be computed based on the pairwise cosine similarity. Specifically, an example is given as follows
The pair with the highest similarity column "PostId" from Table Comments" and column "PostId" from Table "PostLink" with similarity 0.964
2 The pair with the 2nd highest similarity is column "PostId" from Table PostHistory" and column "PostId" from Table "PostLink" with similarity 9.0 3 The pair with the 3rd highest similarity is column "PostId" from Table Comments" and column "PostId" from Table "PostHistory" with
similarity 039
4 The pair with the 4th highest similarity column "PostId" from Table PostHistory" and column "PostId" from Table "Vote" with similarity 0.928
5 The pair with the 5th highest similarity is column "PostId" from Table PostLink" and column "PostIa" from Table "Vote" with similarity 0.917
27

---

Published as a conference paper at ICLR 2025
6 The pair with the 6th highest similarity is column "PostId" from Table Comments" and column "PostId" from Table "Vote" with similarity 0.897
7 The pair with the 7th highest similarity column "ExcerptPostId" from Table "Tag" and column "WikiPostId" from Table "Tag" with similarity
8 0.890
Moreover, we would like to claim that the information provided by Deepjoin is noisy. For example, on the MAG dataset
1 The pair with the 1st highest similarity is column "paper_cite" from Table "Cites" and column "paper_cited" from Table "Cites" with similarity 0.885
2 The pair with the 2nd highest similarity is column "paper "paper_cite" cite" from Table "Cites" and column ppaper_rrieer from Table "Writes" with similarity 0.840
3 The pair with the 3rd highest similarity column "paper Table "Cites" and column "paper writer" from Table "Writes" with similarity 0.832
"paper_cited" cited" from
4 The pair with the 4th highest similarity is column "paper cited" from Table "Cites" and column "paper_name" from Table "HasTopic" with similarity 0.822
5 The pair with the 5th highest similarity column "paper_cited from Table "Cites" and column "paper _ame__aae" from Table "HasTopic" with similarity 0.806
The "paper-writer" wwrterr refers to the author of the paper, which is different from the "paper.cite" which refers to cited papers. However, LLMs can recover from such noise and generate robust results.
E
MORE EXPERIMENTAL RESULTS
E.
RESULTS OF OTHER BASELINE MODELS
E.2 INFLUENCE OF DIFFERENT LLMS
After experimenting with several different language models, we found that some models either com- pletely fail to function, such outputting augmentations of the examples provided in the prompt, or produce similar results. Specifically, we discovered that models like Sonnet 3.5 and those stronger than Sonnet 3.5, such as Sonnet Opus, can serve as effective model backbones. In contrast, models like Llama 3.1 70B (Dubey et al., 2024) and Mistral 2 cannot produce valid results. If we infer based on ChatbotArena (Chiang et al. 2024 performance, we deduce that models stronger than Sonnet 3.5 serve effective backbones.
E.3 DESIGN OF SCHEMAS
This section details the AutoG and expert schema design for each dataset we propose.
E. IREE-CIS
IEEE-CIS is a dataset with weak network effects. Specifically, we find that only two columns "billing region" and "billing country" can bring limited performance gain. However, AutoG-S and experts can't find such relations. AutoG-A outputs the original schema as the best output schema.
E.3.2 ROTKIL
For Retailrocket dataset, AutoG doesn't find the relationship between Category and Item Category table. In this case, achieves sub-optimal performance compared to the expert ones.
28

---

Published as a conference paper at ICLR 2025
Table 6: Experimental results on more backbone models.
TabGNN
Original schema
JTD schema
AutoG
Expert schema
Dataset
Task
Backbone
(Value)
(Value)
(Value)
(Value)
(Value)
Datasets with single downstream task
IEEE-CIS
N/A
GAT
74.65
87.23
87.23
87.23
87.43
HGT
75.82
89.97
89.97
89.97
87.42
PNA
75.49
89.14
89.14
89.14
86.66
RetailRocket
N/A
GAT
81.92
50.13
50.63
79.18
82.84
HGT
83.25
49.06
49.92
71.37
84.95
PNA
82.99
50.43
50.95
82.59
84.27
Moviclens
N/A
GAT
54.78
56.42
62,98
74.81
75.96
HGT
64.12
60.12
63.46
74.42
74.63
PNA
63.23
62.66
62.20
74.33
74.75
Outbrain
N/A
GAT
62.44
52.54
52.73
61.57
63.08
HGT
62,58
52.13
52.78
61.83
63.22
PNA
62.63
52.74
52.98
61.75
63.23
AVS
N/A
GAT
55.18
48.08
54.02
56.19
$5.27
HGT
52.97
49.58
53.12
54.25
56.03
PNA
52.78
48.72
54.19
55.45
55.06
Datasets with multiple downstream tasks
Venue
GAT
44.39
47.98
47.65
$1.08
$1.19
MAG
HGT
45.78
48.24
46.78
46.92
46.92
PNA
46.60
46.25
47.36
51.59
$1.59
Citation
70.92
68.23
79.45
HGT
69.95
67.73
79.31
79.05
78.96
PNA
70.31
65.08
77.45
77.33
77.16
Year
54.27
54.32
54.18
35.23
HGT
43,94
47.12
52.18
53.47
36.73
PNA
37.85
49.75
51.26
51.68
32.39
CTR
50.15
68.65
50.00
73.60
Diginetica
HGT
52.34
65.32
49.85
67.33
67.33
PNA
49.88
66.43
50.15
70.15
70.15
Purchase
7.65
37.42
37.42
HGT
4.61
5.67
9.85
22.07
22.07
PNA
5.33
8.05
18.52
37.58
37.58
Churn
78.04
80.96
85.43
77.77
86.45
Stackexchange
78.63
76.03
87.51
86.70
PNA
78.55
82.92
85.63
93.34
86.64
Upvote
85.96
86.54
88.53
89.00
88.53
84.91
85,35
88.17
PNA
85.72
86.54
88.97
88.77
88.96
29

---

Published as a conference paper at ICLR 2025
--0 77
set 57 be coene
Turida 20
Sartida
der
cn und 10000
Figure 5: Schema for the AutoG IEEE-CIS dataset
Figure 6: Schema for the expert IEEE-CIS dataset
Categey
WEDHA
View
PRatarTs DATE-PPE PC
-Viltand CHAR22)
Wakar
VidTand VACIIN
CINMEAS SOLOITT
wwr_eveladee CHARXZ
Oh Hamm
ha
TlanPrpertt
atron
CHAR322)
Smatanns: CATETING
cateper: CHAR331
Calapersic
SNITT CHAITO)
Figure/ 7: Schema for the AutoG Retail- Rocket dataset
Figure 8: Schema for the expert Recail dataset
30

---

Published as a conference paper at ICLR 2025
E.3.3 MOVIELENS
Compared to the expert schema, AutoG makes the following variations: It builds relationship between the user in Tags and Ratings. 2. It identifies the tag column as text type instead of the category type.
Kattoge
leggoerr mesg
.Ew.eD (CAACT))
cWing WATTA
DNerRD CHAPUST
rNongE CHIRPA
TENNSHIN BNTEINN
realeD CHARCE)
CCHAL
Tay WORCHER
mex
Trastiin DATETIN
Figure 9: Schemn for the AutoG Movielens dataset
Figure 10: Schema for the expert Movielens dataset
E.3.4 OUTBRAIN
For Outbrain, the difference between AutoG and expert schema lies in utilizing the User dummy table. From the experimental results, we find that such a dummy table presents limited influence on the final performance.
ANC
Lomale SPEEI
LacLi Vebuee
Figure 1I: Schema for the AutoG Outbrain dataset
Figure 12: Schema for the expert Outbrain dataset
E.3.5 AVS
Compared to the expert schema, AutoG doesn't establish a relationship between customers. Instead, it introduces new dummy table Brand.
31

---

Published as a conference paper at ICLR 2025
CAARIS
CRABIO
Figure 13: Schema for the AutoG AVS dataset
Figure 14: Schema for the expert AVS dataset
E.3.6 MAG
For MAG, AutoG establishes the same schema for venue prediction and citation prediction task. For the year prediction task, AutoG-S establishes a schema getting around 50% accuracy, while AutoG-A finds that the original schema is the best since there's s limited network effect.
Pager
ININNTO CHARETTI
FERERLATee CHAPARI
-Tabe SANNC MAR
INONCLcee CHAR122)
papan CHANLSS
your
ydar WACHAA
Fend chate Crolk paper CHrAA 30)
Ratcorttudy
Wilte
Wrbte
Authar
IRARRUSE: CHAR22
Hulteer EmArr.S
Irathtzton CHAR132)
-10: C0W---
Figure 15: Schema for the AutoG on MAG dataset, venue prediction/citation prediction task
Figure 16: Schema for the expert MAG dataset
E.3.7 STACKEXCHANGE
For Stackexchange, change the original userid corresponding username in the augmented table to improve the augmentation difficulty. As shown below, AutoG doesn't t establish this relation after the augmentation. However, from the experimental results, we that
E.3.8 DIGINETICA
The expert scheme construction of this dataset has some problems, since some relations don't don't meet the strict PK-FK relations. AutoG can identify this problem, but since the codebase of 4DBInfer can handle such case, stick to their implementation. The relation between the CategoryID column a pitfall. AutoG-S directly generates this relation, and we use AutoG-A to remove this harmful relation.
32

---

Published as a conference paper at ICLR 2025
APP MindARt
(opeecte (HARTZZ
GRRAY
2apper, the CHARD32
Fall Wtuatyyyyyaoo.o..o...
VIV
fee CHANE52)
Hol Siger
Ryaar: VARCHAR
wiTer
ardty
papAY wntter WARHIA
CHANGR
Figure 17: Schema for the AutoG-S on MAG dataset, year prediction task
Figure 18: Schema for the AutoG-A on MAG dataset, year prediction task
Figure 19: Schema for the AutoG Stackex-
Figure 20: Schema for the expert Stackex- change dataset
change dataset
33

---

Published as a conference paper at ICLR 2025
Har yoar DIN011
Figure 21: Schema for the AutoG Diginetica dataset
Figure 22: Schema for the expert Diginetica dataset
E,4 DESIGN OF SYNTHETIC DATASETS
The schema design for Section 5.3.2 are shown in Figure 23 and Figure 24
CUAPS
THI
A7.04: VARCHAR
Hame WERTHAE
Hlsation MWRCIAR
URgSt
Firvet CHAMESOO
EET TE BRRAY
2A5AA VARCHAR
rezerD CHARST
Tall
Eipether
Macdealteeer. VMCCIA
Powil CHANRT99
NolEp VARL.AR
CSSAN 75ETm CHARCS
Figure 23: Schema for augmented MAG dataset
Figure 24: Schema for anonymous aug-
mented MAG dataset
34


Reviews:
[{"review_id": "QkhR96lyio", "reviewer": "Reviewer_oErT", "summary": "This paper proposes a method to convert table data into a graph by leveraging a large language model (LLM). \nThe primary features are as follows:\n- Detailed design of LLM prompts.\n- Comprehensive definition of basic actions (Section 4.2).\n- Quantitative oracle using GML, potentially effective for this task (Section 4.3).", "questions": "Questions\n- Is the problem defined as follows?  The input: relational data, GML for downstream tasks, and records and class labels for training and validation data. output: transformed graph data from the relational data.\n\nComments\n- The term \"RGAT\" may be incorrect and should be \"GAT.\" The cited paper (Veličković et al., 2017) refers to GAT.\n- Using GML models suited for homophilic graphs (e.g., RGCN, GAT) as an oracle may be unsuitable for tasks on heterophilic graphs. This issue could be addressed by selecting GML models tailored to heterophilic graphs as the oracle.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- S1: The proposal introduces a carefully devised LLM prompt (Appendix D)\n- S2: The evaluation experiments show that the proposal significantly outperforms existing techniques and achieves results close to those of manual graph generation.\n- S3: Section 3.1 identifies C4 (graph variations) , which is indeed an important challenge.\n- S4: Generating co-author relationships as edges is definitely effective for node classification of homophily graphs.", "weaknesses": "- W1: The problem definition is not explicitly stated. \n\n- W2: Although the proposal utilizes carefully designed LLM prompts, it relies on standard techniques in LLMs like few-shot learning and chain of thought (CoT), making the novelty unclear.\n\n- W3: While the quantitative oracle evaluating GML with a validation set seems effective, Table 4 suggests the oracle may not be essential. The GML results are highly dependent on the choice of label selection (e.g., venue vs. year), making the conclusion \"LLMs can generate good candidates merely based on prior knowledge\" is not reasonable. We encourage the authors more detailed discussions here.\n\n- W4: Concerning the five challenges in Section 3.1, if the schema is normalized, issues like C2 and C3 (1NF) seem unlikely to arise. Additionally, the proposal only addresses node classification and does not handle tasks like link prediction or node clustering, so C5 appears to be an overstatement.\n\n- W5: There is a lack of evaluation on speed improvements. Section 4.2 claims the high cost of JTD, and Section 4.3 mentions a design for potential speed-ups, making such an evaluation essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ll51DGyfbq", "reviewer": "Reviewer_4sTF", "summary": "The authors provide a method for automatically converting relational tables into a knowledge graph. They forego manual engineering by using an LLM approach. First, the LLM is instructed to create a schema, restricted by calling certain functions. Then, an oracle is used to provide feedback for this schema generation. That feedback is then again used by the generator.", "questions": "* Recently, Alivanistos, et al. (2024) presented \"The Effect of Knowledge Graph Schema on Classifying Future Research Suggestions\". That work suggests that the classification performance depends on the chosen schema. Is this the same you observed in the current work? Should one consider existing schemas or let the model figure this creation out as well?\n* Much of the idea is clearly supported by argument, the choice of the oracle as a discriminator and the heuristics one gets out of that are rather arbitrary. It is not clear whether these are the best options. How do you know?\n* You only use RGCN and GAT, both of which are rather GNN based learning methods. Would you expect other methods to behave similarly? Why?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The general observation is an important one: the conversion of tabular data into graph form cannot be taken for granted. Existing graph benchmarks based on tables avoid the hard cases.\n* The setup with an LLM to generate candidates is pretty nice. Restricting its generation freedom using function calls is a good idea as well.", "weaknesses": "* It remains unclear how well this method performs on the long tail. The evaluation is averaging over many cases, but the results might be dwarfed by very common ones.\n* The paper only considers very well behaving rectangular tables (relational database style), with column names and all. Even the data types are given. There is also a lot of web tables around with large datasets available. One could certainly question whether te chosen setting is realistic, now.\n* The datasets are pretty small.\n* Citations for relational graph learning are to 2023< papers, while this has been studies for 10+ years.\n* It is not clearly argued why relational graphs are most suitable. Other formalisms like hyper-relational ones might be more appropriate.\n* I suggest removing the claim of surpassing human experts. It is not clear whether all schemas from the experts are  such that they were made as an ideal schema for a GNN architecture / the specific task.\n\nminor:\n*  treate each ->  treated each", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6eghc1Jc9", "reviewer": "Reviewer_8qV7", "summary": "This paper focuses on the impact of automatic graph construction from input tabular data on downstream Graph Machine Learning (GML) tasks. It introduces a benchmark for evaluating graph construction and proposes AutoG, an automated graph construction method.", "questions": "1. Can you prove that the conclusions from RGCN and RGAT apply to all GML models?\n2. Have you considered proposing a better metric to evaluate the constructed graph itself? Furthermore, if the constructed graph is closely tied to specific downstream tasks, does that suggest that T2G might be more suitable as a data augmentation module to be explored with specific GML models, rather than as a standalone benchmark?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. In the field of GML methods for tabular data, this paper is the first to concern graph construction evaluation, and introduces a benchmark for it.\n2. The authors identify five key challenges in converting tabular data into graphs and proposed AutoG, an agent-based approach. They design actions for the large language model (LLM) to utilize its prior knowledge for augmenting graph construction. Additionally, they implement a feedback mechanism to calibrate the LLM's output based on the validation performance of downstream GML tasks.\n3. The paper is well-structured, clearly expressed, and includes detailed information.", "weaknesses": "My main concern is with the \"metrics used in the proposed benchmark for evaluating the conversion of tabular data into graphs (T2G).\" As stated in line 216, the authors use the performance of fixed GML models (RGCN, RGAT) trained on the generated graphs for quantitative evaluation. However, as a benchmark for T2G, it should be independent of specific downstream GML methods. Thus, the evaluation is limited to these two models and lacks generalizability to all GML models. Until it is demonstrated that the conclusions from RGCN and RGAT apply to all GML models, both the benchmark and the experimental findings in this paper remain constrained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sv7kpVOOmI", "reviewer": "Reviewer_fBje", "summary": "The paper introduces a benchmark for constructing graphs from tabular data as well as an LLM-based method that generates graph schemas without human intervention.", "questions": "The authors should carefully address the concern about the relationships with previous database research (W1, W2) and the assumptions of the method (W5). Moreover, the inclusion of a larger, more challenging set of datasets would also be encouraged (W4).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(S1) The paper proposes a direction to tackle the interesting problem of transforming tables into graphs.\n\n(S2) The benchmark datasets are a valuable addition to the community.\n\n(S3) The paper performs several experiments to show the effectiveness of the method on various data.", "weaknesses": "(W1) The paper does not consider a rich database literature in mining functional dependencies [1,2] and data profiling [3] that automatically extracts dependencies among table columns. These methods can be used to generate more refined schemas that can solve some of the challenges expressed in Section 3.1. This method should be part of the experimental evaluation.\n\n(W2) After having mined uniqueness constraints and functional dependency, one can apply conventional data normalization to further remedy the problems highlighted in challenges C2, and C3. From this standpoint, it is not clear why a mine-and-normal step would not solve most of the issues.\n\n(W4) The benchmark only contains fairly small datasets with, at most 8 tables, for example, PTE has 38 tables (https://relational-data.org/dataset/PTE). There are also several datasets here: https://relational-data.org/\n\n(W5) Another challenge, not specifically addressed by the current paper, is missing values and NULL values. Can the proposed method cope with those? If not, why?\n\n(W6) Clarity: The paper is, in many parts hard to follow. Here are some examples\n\n- Figure 2: It is not clear what flexible design is and what the prompt looks like\n- at least two columns as FK and no PK: provide an example\n- chain of augmentation: please state what it is\n- validation set performance: explain\n\n(W7) 488: How does the method ensure that the test data is not included in the pre-training of the LLM? Do you have access to the training data?\n\n(W8) The paper relies on LLMs that are subject to changes. Does it mean that one has to adapt the method to a different LLM every time?\n\n### Minor\n\n(M1) I encourage the authors to use citations from published sources while in the current draft, several citations are from arXiv\n\n(M2) Typos:\n\n- 133: Treate\n- 277: LLM generates / tends → LLMs generate / tend\n- 282: single-steply\n\n(M3) In modelling the graph $G={V, E}$ it seems that the nodes are just a set of types as $\\mathcal{V}=\\bigcup_{v\\in V} \\mathcal{V}^v$. This definition does not seem standard for Heterogeneous graphs. There should be a set of nodes, each with a type (or a set of types) and a set of edges among the nodes.\n\n[1] Yao, H. and Hamilton, H.J., 2008. Mining functional dependencies from data. *Data Mining and Knowledge Discovery*, *16*, pp.197-219.\n\n[2] Liu, J., Li, J., Liu, C. and Chen, Y., 2010. Discover dependencies from data—a review. *IEEE Transactions on Knowledge and Data Engineering*, *24*(2), pp.251-264.\n\n[3] Abedjan, Z., Golab, L. and Naumann, F., 2015. Profiling relational data: a survey. *The VLDB Journal*, *24*, pp.557-581.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)