Retrieve, Merge, Predict: Augmenting Tables with Data Lakes
Authors: Riccardo Cappuzzo
Aimee Coelho
Félix Lefebvre
Paolo Papotti
Gaël Varoquaux
Venue: Accepted by TMLR ()

Abstract: Machine-learning from a disparate set of tables, a data lake, requires assembling features
by merging and aggregating tables. Data discovery can extend autoML to data tables
by automating these steps. We present an in-depth analysis of such automated table
augmentation for machine learning tasks, analyzing different methods for the three main
steps: retrieving joinable tables, merging information, and predicting with the resultant
table. We use two data lakes: Open Data US, a well-referenced real data lake, and a novel
semi-synthetic dataset, YADL (Yet Another Data Lake), which we developed as a tool for
benchmarking this data discovery task. Systematic exploration on both lakes outlines 1)
the importance of accurately retrieving join candidates, 2) the efficiency of simple merging
methods, and 3) the resilience of tree-based learners to noisy conditions. Our experimental
environment is easily reproducible and based on open data, to foster more research on feature
engineering, autoML, and learning in data lakes

Published in Transactions on Machine Learning Research (5/2025)
Retrieve, Merge, Predict: Augmenting Tables with Data Lakes
Riccardo Cappuzzo
riccardo Srcando.cappuzzo@inntt
SODA Team Inria
Saclay
Aimee Coclho
Sante..ccllooSdaaauuuu.. com
Dalaiku
Poris
Felix Lefebvre
felin.lefebure@Bnnrii..
SODA Team Inria
Saclay
Paolo Papotti
papotti@euuecom.f
EURECOM
Biot
Gael Varoquaux
gael wwr.uuorrpuuunnninrii..
SODA Team Inria
Saclay
Reviewed OpenReview: https: https:://pperreviee openrevieww net/ forum? id=4uP_N6yf11
Abstract
Machine-learning from disparate set of tables, a data lake, requires assembling features by merging and aggregating tables. Data discovery extend autoML to data tables by automating these steps. We present in-depth analysis of such automated table augmentation for machine learning tasks, analyzing different methods for the three main steps: retrieving joinable tables, merging information, and predicting with the resultant table, We use two data lakes: Open Data US, a well-referenced real data lake, and a novel semi-synthetic datasot, YADL (Yet Another Data Lake), which we developed as a tool for benchmarking this data discovery task. Systematic exploration on both lakes outlines 1) the importance of accurately retrieving candidate tables join, 2) the efficiency of simple merging methods, and 3) the resilience of tree- based learners to noisy conditions. Our experimental environment is casily reproducible and based on open data, to foster more research on feature engineering, autoMT, and learning in data lakes
1 Introduction: learning on data lakes needs feature engineering
New tools keep facilitating data science with machine learning, partly automating it, with autoML HHe
et al. 2021 Karmaker et al. 2021 Erickson et al. 2020 Feurer et al. 2022 Hutter et al. 2019). However, they typically take a single table as an input, while data scientists often start from data lake: loose corpus of tables (Nargesian et al. 2019 Spotify 2020 Phan 2023). Learning then requires assembling multiple tables, which needs either in-depth knowledge of the data, or data discovery (Fan et al. 2023a Hulsebos et al, 20244.. Data assembly typically studied in scenarios where the schema. known (Robinson et al. [2024 Wang al. 2024h: how columns of various tables are related and which can be merged, in relational database. However, data lakes often come without this information. In this work, consider data lake

---

Published in Transactions on Machine Learning Research (5/2025)
situation where the meaningful merges are unknown and where only a fraction of the tables in the data lake are relevant for the given machine learning (MI.) task. Consider the following example scenario:
Alice, a data scientist, wants to predict the rating given to movie: she has access to a table with information about movies (e.g., year of release, director, language, budget, ...). She wants to complete the table, finding more information on the subject beneficial to her task; for ezample, joining the table about movies with a table on the cast of those movies might help, as some actors tend to appear in movies with better ratings. Alice has access to some large repository of data, or to some search engine that she can query to find additional tables (Castelo et al.) 2021: Google, 2023). However, she does not know the schema describing the relation between tables in the data repository. Her objective is to find the tables that are most relevant to her task, to merge them with the original lable and finally to use the improved table to build model that predicts the movie rating. To ensure reliable prediction results, a cross-validation setup repeats these steps over different train-test splits; as resources (compute, RAM...) are limited and rross-validation has a cost, Alice is interested in finding the optimal set of candidates to achieve the best results within certain budget.
Research in this area spans the database and machine learning communities. The corresponding pipeline
(Figure D uses retrieve, merge, and predict steps that involve four tasks: 1) retrieving tables that are
joinable with the original table (Fernandez et al.) 2019 Zhu et al.
2021 Fernandez ett al. 2018 Dong et al. 2023 2) selecting which joins should be executed to improve the performance of
2019
2016
Castelo et al.
the subsequent ML model [Esmailoghli et al. 2021 Liu et al. 2022
Galhotra et al. 2023 Deng et al. 2017a),
3) aggregating results in cases of one-to-many or many-to-mant joins (Chepurko et al. 2020 Kanter &
Veeramachaneni 2015p, 4) predicting with supervised learning on the resulting table, a tabular learning
problem (Shwartz-Ziv & Armon 2022 Borisov et al., 2022 Grinsztaju et al., 2022). Elaborate methods have been designed for each of these tasks separately. Yet, how they contribute to the overall machine learning pipeline is often unclear. Prediction performance is not evaluated consistently in the database literature. Moreover, evaluation on data lakes is particularly challenging. First, there is no openly available reference data suited for this purpose - both a data lake and tables with corresponding analytic tasks are needed. Second, solid evaluation must run the pipeline many times across loosely structured, messy eusembles of tables. tfate-of-the-art publications relevant to this pipeline seldom come with implementations robust and scalable enough for cross- valssvvaliotion loops involve many complex elements, such as probabilistic data structures large language models, which need careful software engineering to be production ready. The difficulty of operating those complex pipelines does beg the question of when the prediction gain is worth the operational cost. State-of-the-ar research often
data lake. Indoed, data discovery and assembly code
YADL
Artet, Authored Movie, Director
Movie
Year
Rating
Rating
Director
Country
Country: Pepulation
E.T.
1982
7.9
E.T.
1982
7.0
Spieterr
Stower
USA
Companyy Locosedin
Tay Story
1995
Story
John
USA
Indexing
Retrieve
Merge
MinHashh
Exact
Join Candidates
Select
Aggregate
RidgeCV
Predict
Hybrid MH
Starmie
Morke Seaeeyy
Movie, Director Mirin, Country
Highest Cont, Join Best Single Join
Top-k Full Join
Mean
Any
Augmented table
Calboost
RealMLP
ResNet
Index
Step. Greedy Join
Index
Data preparation
Data usage
Figure The evaluation pipeline. Given a base table, the three main steps (Retrieve, Merge, Predict) augment it with the information from the lake to improve the prediction performance. The data preparation step can be executed offline, and the resulting index may be reused across different data usage instances.
2

---

Published in Transactions on Machine Learning Research (5/2025)
leaves aside the question of complexity, which involves not only computational cost, but also preparation and maintenance burden [Seulley et al. 2015 Paleyes et al. 2022.
We study the full data-discovery for machine learning pipelines, with an eye on exploring the complexity gradient in a reproducible way. For this, we contribute Yet Another Data Lake (YADL), a dataset that enables systematic exploration of the aspects of the pipeline important for good prediction. Built. from the YAGO knowledge base (Mahdisoltani et al., 2014), YADL provides a controlled environment to test methods. To analyze the discovery and merge pipelines across different degrees of complexity, YADL's synthesis procedure allows us to vary the shape (number of rows and columns) and degree of redundancy of the resulting tables.
We investigate the impact of the main steps on the end goal of learning from data lakes: preparing the data structures needed to search for candidates, retrieving and selecting potential augmentation candidates, integrating them with the base table, and subsequently evaluating learning on this assembled data. Considering both retrieving and combining tables in a data lake and the downstream learning task, our work bridges database search and machine learning, a topic of much interest (Fey et al.) Robinson et al.] 2024 Cvetkov-lliev et al. 2023 Wang et al., 2024). It has two main objectives: first, determine which pipeline steps are most important for prediction performance, and second, develop level playing field that facilitates a holistic assessment of learning on data lakes.
We analyze various methods in the four tasks, with an exhaustive empirical evaluation that required approximately 21 years or 189k CPU and GPU hours (Table 7). Our findings show that 1) automated selection of joins is noisy, creating very messy tables for which tree-based methods are more robust than deep learning models; 2) better retrieval of joins reduces the amount of noise and the fraction of null values; 3) relying on Jaccard containment (i.c., the fraction of entities of the table to augment found in a join candidate) as criterion for relrieving joins improves downstream prediction performance; 4) downstreem performance plateaus rapidly as more tables are merged, while the resource cost increases; 5) finally, simpler, metric-based join-discovery methods outpace their complex counterparts in terms of speed and the larger cost of more elaborate methods does not yield proportionate gains in prediction performance.
To validate the generality of our findings, we contrast the insights obtained from the analysis carried out via. YADL with those obtained from Open Data US, well-referenced real data lake (Nargesian al. 2018
Galhotra et al. 2023): : this comparative approach ensures that YADL is representative enough to be benchmark. We consider 6 input base tables from the literature which pertain to multiple domains.
After outlining the problem setting and previous work in Section 2 we share our main contributions:
1. We develop YADL: novel benchmarking data lake that allows to test retrieval and augmentation techniques in a controlled environment (Section 3). YADL, the base tables, and the pipeline are available and easily extendable to spur further research.
2. We implement a full prototype pipeline for augmenting tables from data lakes (Section 4h. We then measure the prediction performance, execution time, and RAM usage of each method.
3. We conduct an experimental study on 6 base tables over various data lakes, retrieval techniques, join selection methods, aggregation solutions, and ML models (Section 5). Results provide insights on the steps of pipeline learning from data lakes.
We conclude the paper with a discussion of future research directions in Section 5
2 Problem setting and related work
Problem setting. Consider user training a ML model to predict some quantity. The corresponding quantity appears in the training data as a target column Y of a base table T. Assume our user has also access lo a large collection of tables (a dola lake) D = (TT,,, ..,,
J some of which may contain additional information that can enrich the base table T. The data lake may be publicly available (Castelo et al.)
2021 Mattmann et al. 2018 Srinivas et al. 2023 Hulscbos et al. 2023 Nargesian et al., 2018) or private (e.g., corporate data). Departing from many studies on learning on relational databases, where schemas are provided (Motl & Schulte 2015 Fey et al. 2023 Robinson et. al. 2024), we consider unstructured data lakes,
3

---

Published in Transactions on Machine Learning Research (5/2025)
i.e, data lakes without schema to specify any PK-FK (Primary Key- Foreign Key) relationship between the tables.
Each of these tables T3 is a bi-dimensional collection of data organized in columms Cj E T' that may include categorical (names, codes etc.), numerical data (price, revenue, tax rates etc.) and text (product descriptions, user reviews etc.). While cross-table metadata such as foreign keys is not available in this setting, joining T with some of the tables in D would be beneficial for the target prediction task.
Given T and D, table Ta D is considered to be joinable, or join condidate for T on query column Q E T, if at least one of the columns Cg E Ti has a non-empty intersection with one of the columns in table T: if EQ E T, 3C2 E Tie QQC + 0.
The user wants to optimize the performance of the ML model on a collection of columns (or "features") X according to the quality of the prediction for the target Y (e.g., the movie rating). Columns in X may come either from T, or from joined tables in D.
Furthermore, consider use case where this operation pipeline is wrapped in cross-validation setup to ensure the reliability of the results: describe this use case as "Data usage" in Figure 1
Different steps. In such scenario, the user is likely to rely on the following main steps: Join Candidate Retrieval, Join Candidate Merging (which includes the Join Selection and Aggregation tasks), and Prediction. Depending on the specific scenario, some of the tasks may be executed in a different order or not at all. In the following, we will drop Join from the names when clear from the context.
Another related problem, which do not study, is that of finding new samples rather than features (called "table unionability" in databases): identifying which tables may be "appended" to T to increase the number
of rows (Khatiwada et al. 2023a Fan et al. 2023b Nargosian et al. 2018).
Finding the join candidates Given a base table T and a data lake D, the Candidate Retrieval task consists in discovering join candidates (Retrieve in Figure I). This task looks for tables that can be considered as "candidate joins" for the given base table. Different moth have been developed for different.
scenarios: some "dataset search engines" tackle internet-scale crawls of tables (Castelo et al. 2021 Mattmann ct al., 2018); other methods are instantinted data stored locally (Zhu et al. 2019 Fernandez al. 2019
Zhu et al. 2016 Fernandez et al. 2018 Khatiwada et al. 2022 2023b)
While integrating data is not a central focus of the ML literature, augmenting features via joins is recognized as key to ML (Paleyes et al. 2022 Kumar ct al. 2016). Still, the blind addition of features may lead to diminishing or negative returns, which led to the development of systems that seek to integrate only correlated
features (Esmailoghli et al. 2021 Santos et al. 2021). Alternative systems soek to augment T by senthesizing
new features (Kanter & Veeramachaneni] 2015 Cvetkov-lliev al.) 2023 Zhao & Castro Fernandez 2022).
Recent contributions focus building graph representations of relational databases to leverage GNNs for
learning across tables (Fey et al. Robinson et al. 2024 Wang et al. 2024). An orthogonal research direction
is concerned with the problem of scarch and augmentation while maintaining privacy (Huang et al. 2023a by.
Though implementations may vary, retrieval methods used to find join candidates share some similarities 1) they involve an offline phase to either index the data lake (Zhu et al. 2016 Fernandez et al. 2019), or train model (Fan et al. 2023b Dong et al., 2023), and 2) they rely on similarity metrics across columns to select candidate joins. For our experiments, we assume the user has run the oflline step prior to the retrieval operation (the "data preparation" section to in our pipeline, Figure I).
Often, the similarity metric is Jaccard Containment (JC), defined as
IQn C QQ
Jaccard Containment :=
(1)
where Q E T is a query column in base table T, Cj E D is a candidate column in table Tg, IQ n Cf is
the cardinality of the intersoction between the two sets, and |Q] is the cardinality of the query set itself. Intuitively, this ratio is high, then a large fraction of Q found in Cj suggesting that the two columns should be joined. Approximate methods scale JC-based retrieval to very large databases (Zhu et al. 2016
4

---

Published in Transactions on Machine Learning Research (5/2025)
Beyond JC, other metries have been used, such as top-k set similarity Zhu et al. 2019), combinations of JC and embeddings-baae based similarities (Fan et al. 2023b or other embeddlings-baset metrics (Cong et al.) 2022 Habibi et al. 2020), such as approximate nearest neighbor search (Dong et al.) 2023 on model embeddings. Methods based on Jaccard similarity are vulnerable to typos, which led to the
Fernandez et al. 2019:
2018).
development of methods that perform fuzny or semantic matching such as Fan et al. (2023b); Mundra el al.
Dong et al. (2017b); Abedjan et al. (2015); in this work, we focus on scenarios that do not involve fuzzy matching and thus need to rely on an exact metric to ensure overlap between columns, such as JC.
(2023): ;
Retrieval methods are often designed for large data lakes (up to millions of tables Zhu et al. 2019 Fan
ct al. 2023b) and to maximize recall. However, three issucs arisc. First, these methods do not assess the relevance of join candidates to the downstream task. While a large containment value may indicate that a join can be done, it provides no guarantee that this join is actually useful (Kumar et al. 2016). Second, the number of candidate joins could become too large for practical use. Manually identifying the best candidates is time-consuning, and performing all joins might be too expensive in terms of time/memory constraints (Santos et. al. 2021). A user-defined threshold on the containment, can filter out the least promising joins, but. deciding the correct threshold is problematic; alternatively, it is possible to select only the top-k candidates by containment. Third, Jaccard containment does not take into account the cardinality of column: in an extreme case, if a column Q E T contains only single value, it would have perfect overlap with any column C% E D that contains the same value. While this behavior may be desirable for some specific retrieval tasks, it further expands the number of candidates with high containment. The presence of duplicate tables worsens each of these issues by introducing potential false positives.
To make retrieval readily useful, the Join Selection task identifies subset of candidate joins that marimizes the prediction performance over a downstream task (Select in Figure 1p. These methods use various strategies
to add value to retrieval: profiling candidate joins according to various metrics (Galhotra et al. 2023 Flores
et al., 2021), rules to remove joins that are not useful (Kumar et al.
2016 Shah et al.) [017]). joining over
sketchos or coresets of the data (Wang ct al. 2022 Santos et al. 2021
: or executing each candidate join to
find those that bring benefit (Esmailoghli et al. 2021 Chepurko el al. 2020 Liu et al. 2022 Galhotra el al.j
2023 Gong et al.) 2023 Dong& & Oyamada 2022). In this work, we focus on the latter, very popular, strategy. We consider ML models that operate rows as samples, columns forming features. Augmentation enriches features while keeping the original set of samples, thus requiring left join. A challenging consequence is that un-joined rows in the left. table will lead to missing (null) values in the new columns.
We split Candidate Join Retrieval and Selection to tease out the retrieval metric from the benefit for the task: the first measures similarities between columns, while the second depends on the information in the foreign table.
Merging the candidates After the set of join candidates has been retrieved, they must be merged with the base table to augment it with their columns. It is not always possible to limit joins to one-to-one relationships; often, we join on one-to-many or many-to-many relations. For example, to join a table about movies with one that contains movie ratings on column "movic title", every movie with more than one rating is in a one-to-many relation, thus the content of all the rows in such relations gets duplicated Sample duplication is problematic when the downstream task involves a ML or statistical analysis method, as each row corresponds to one sample.
The Aggregation task bridges the result of the join with the downstream methods, combining the information contained from a potentially large number of rows into one (Aggregate in Figure 1). More precisely, given a tuple (row) t in the base table T' that needs to be joined with n tuples from a table (A, B) over A, the goal is to augment t with attribute B by selecting one value that represents the information from the n joining tuples. How to select a representative value for the new altribute reminds data integration problems such truth discovery (Dong et al. 2009) or data fusion (Beihholder & Naumann 2009). In this spirit, Deep Feature Synthesis (DFS) (Kanter& & Veeramachaneni 2015) takes set of tables and join plan to recursively aggregate replicated instances using functions such as average, median, and mode.
More detail provided in Appendix subsection B.3
5

---

Published in Transactions on Machine Learning Research (5/2025)
Depending the join selection strategy, aggregation may be executed during the selection process, or after the set of candidates has been chosen. In any case, it is paired with the selection procedure by the need of training the ML model on the augmented tables. To represent this coupling, we combine the two tasks in a single Merge step in Figure
Learning on the augmented data Finally, the integrated table is used to train a model to Predict a larget variable through supervised methods. Tabular machine learning is the subject of very active research, with well-established tree-based methods (Grinsztaju et al.] 2022 Prokhorenkova et al. 2018 and rapidly
progressing tabular deep learning (Holzmüller et al.) 2024 Ye et al. 2024 McElfresh et al., 2023 Hollmann
et al. 2022 Hegselmann et al. 2023 Arik & Pfister 2021 Gorishniy et al.) 2024h. It is not possible in our study to explore all recent. methods, in particular given that some incur compute costs incompatible with our extensive evaluation and the combinatorics of pipelines. We consider three broad range of methods: 1) linear regression/ classification (as implemented in scikit-learn Pedrogosa et al.] 2011), , 2) treo-based methods such as Cal (Prokhorenkova et al., 2018). : 3) or deep learning methods well suited for tabular data, such as ResNet (Gorishniy et al., (021) or RealML. (Holzmüller et al. 2024). While we present the prediction task as distinct from the prior ones, some pipelines implement learning at intermediate sections of the workflow to ensure the selection of the most appropriate candidate tables (Galhotra et al. 2023 Huang et al. 2023ch.
3 Building Yet Another Data Lake
Yet Another Data Lake (YADL) is a semi-syuthetic data lake built by recombining the data present in the YAGO (Suchanck et al., 2007 Mahdisoltani et al. 2014) knowledge base (KB) P to generate collection of tables. Our goal is to have scalable, high-quality and consistent data lake that allows users to evaluate the main steps of the pipeline in Figure 1 while avoiding some of the confounding factors that come from working with unvetted data, such as typos, inconsistent schemas and data format, and other sources of noise. These challenges can be added to YADL, c.g., by genorating typos in table entrics.
the source of the original data. YAGO (Suchanek et al., 2007) is a KB composed of RDF triplets; each triplet has subject connected to an object through a predicate (or relation). Subjects and objects are considered to be entities, e.g., in the triplet "Paris, locatedIn, France", "Paris" is the subject entity, "France" is the object entity, and "locatedIn" is the predicate that connects them. The object may also be a lexical value such the one for the "population density". Each entity belongs to set of classes (or types), arranged in a taxonomy; "Paris" belongs to the class "City", subclass of "Populated place", itself subclass of "Geographical locations".
YAGO:
From knowledge base to relational tables. Information in data lakes is typically stored as tables, rather than the triplet format of YAGO. For this reason, we first rearrange the YAGO triplets by converting them into binary tables, then we filter and recombine the binary tables to create multiple YADL variants (Base, 10k and 50k) that differ in their size and in the properties of the tables contained therein. YADL variants can be obtained by changing the parameters used for the synthesis, detailed in Appendix A
4 Implementations of the retrieve-merge-preliie pipeline
We now discuss the pipeline steps in Figure 1 For the Retrieve step, we discuss different possible retrieval strategies; for the Merge step, we propose different join selectors and aggregation methods; finally, for the Predict step, we go over the ML methods used to test the prediction performance. We explore each section in more detail in Appendix B.
Retrieving the candidates. We consider retrieval methods that work by taking a query column and return a -possibly ranked - list. of candidates.
We consider four retrieval strategies that explore different approaches and trade-offs. We first distinguish between and "online" methods: methods in the first category require some degree of offline data
We YAGO 3.0.3 (Mahdisoltani al. 2014), which is updated 2022.
6

---

Published in Transactions on Machine Learning Research (5/2025)
preparation in order to be executed, such as constructing data structures, training a specific model; methods in the latter calegory are instead executed directly "online" as part of the "Data usage" " section.
Exact Matching is on "online" method, and measures the eract Jaccard containment between it and every column in the data lake. The "offline category includes MinHashLSHEnsembee (MinHash) (Zhu et al., 2016), Hybrid MinHash (a method we develop); and Starmie (Fan et al.] 2023bp. MinHash builds an index on the data lake, and when queried returns candidates whose Jaccard containment is greater than a certain threshold without ranking them. Hybrid MinHash combines Exact Matching and MinHash by taking the candidates returned by MinHashing and measuring their exact containment, thus limiting the number of columns to be considered and reducing the computational cost of Exact Matching, whilst introducing a candidate ranking. Finally, Starmie builds embedddings for the query column and every column in the data lake using a language model, then combines cosine and Jaccard similarity to rank candidates. All methods share the same drawback of requiring some degree of updating/retraininn retraininn as a data lake evolves over time More detail is provided in Appendix B.1
The "Data usage" part of the pipeline 1 is transparent to the retrieval method: given list of join candidates, it will test each candidate regardless of how the list was constructed. To reflect practical limitations on runtime and compute resources, we introduce a budget constraint on the number of retrieved candidates to evaluate. We choose to keep the top-30 candidates for most of our experiments, as we observe that almost all candidates with high containment are within this limit (Figure 7 in Appendix), it has empirical confirmation in practice [Spotify] 2020). and our experiments show that results plateau past this threshold (Figure 4b. In Section 5 we explore in detail how the value of affects scalability and results.
Merging base table and join candidates. In this step, the retrieved candidates are combined into a new integrated table that joins the information in the base table I" with additional information from augmentation tables. As shown in Figure 1 the merge step combines join selection and aggregation, as both must be executed to build the augmented table used for Prediction.
Join selection We consider two different overarching strategies for selecting candidates: metric-based selectors, which rely on metric to choose which candidates should be joined, and results-based selectors, which instead iterate over candidates and select the best by explicitly executing joins. The first category includes Highest Containment, which ranks candidates by containment, then joins the first, and Full Join, which relics entirely on the ranking provided by the retricval method to join the candidates without filtering them. For the second category, we implement Best Single Join and Stepwise Greedy Join. The first works with one candidate at each iteration, during which it first joins the candidate, then it trains and evaluates prediction model the resulting table; after iterating over all candidates, it selects the single candidate with the best performance. The second method iterates over each candidate like in the previous case, however it retains all candidates that improve the prediction performance, so that the augmented table grows over time as new candidates are joined. Stepwise Greedy Join is related to forward feature selection (Guyon& & Elisseeff 2003), as it greedily adds new tables as features during each iteration, and represents a common approach in the database literature (Galhotra et al. 2023 Chepurko et al. 2020 Appendix B.2 details the implementations.
Aggregation We test. three aggregation strategies: Any selects any row at random from each group of matched tuples; Mean replaces all duplicated numerical (categorical) values by the mean (most frequent) of all values for that attribute in the group; finally DFS (Deep Feature Synthesis (Kanter & Veeramachaneni 2015)) greedily generates new features for each column in the augmented table by aggregating groups of tuples, measuring statistics over the groups (e.g., mean, median, most frequent value), and adding said statistics as features; this is done for every join. We expand on this in Appendix [33
Supervised lcarning with a ML model. The learning step is performed on the final, integrated table obtained by merging the candidates on the training split of the base table. We evaluate four predictors. We use a Ridge regressor/ regressor/classiftee /classifie as a linear baseline using the scikit-learn RidgeCV (Pedregosa et al.) 2011 implementation and default parameters. CatBoost (Prokhorenkova et al.] 2018) is a stato-of-the-art GBDT method, optimized to handle categorical variables. ResNet (Gooishniy et al. 2021 is our bascline neural method. RealMLP (Holzmüller et al. 2024 is a. NN-based method that incorporates number of "good
7

---

Published in Transactions on Machine Learning Research (5/2025)
defaults" to improve performance over standard NNs. For ResNet and RealMLP use the implementation and default parameters provided by pytabkit (Holzmälerr et al. 2024). We do not perform hyperparameter optimization. Further detail on implementation details, pre-processing of the data, and cross-validation setup are provided in Appendix C.2
5
Experimental study
5.1 Settings
Table 1: Statistics for the tables in the data lakes. "N. cols" and "C. cols" refer to numerical and categorical columns, respectively.
For our experimental campaign, we test the different sections of the pipeline over three dimensions: prediction performance, execution time, and RAM usage. Focusing multiple dimensions allows us to have a view of the different trade-offs across methods.
YADL Binary
YADL
OpenData
Base
10k
50k
US
Data lakes We use four YADL variants (Binary, Base, 10k; 50k) (as detailed in Section 3) and Open Data US. data lake employed in the literature (Galhotra al. reported in Table Starmie did not run on YADL 50k (not enough RAM) and Open Data (noise in the data crashed the model), so these data lakes are excluded from figures that involve Starmie. Appendix Figure reeport the measured containment. for the base tables used in our experiments
lables
30072
10059
47223
5591
Size (MB)
629
10051
5557
27718
4062
Tot. rows
20.1M
672M
242M
1.20B
95.7M
Tot. cols
140
95.2k
127k
624k
133k
2023 Zhu al., 2019; 2016); statistics
Avg rows
287k
22.3k
24.0k
25.4k
17.1k
Avg cols
2.00
3.17
12.63
13.21
23.86
Avg N. cols
0.30
0.39
3.54
3.59
11.10
Avg C. cols
1.70
2.78
9.08
9.61
12.76
Avg nulls
0.00
0.31
0.62
0.64
0.09
Base tables We evaluate four tables from sources external to the lakes: 2021 US Road Accidents, and Housing Prices. As "internal" tables, derived from lakes, have been used in several previous works for evaluation (Galhotra et al. 2023 Chepurko et al.] 2020). we also include one per lake: US County Population (from YADL) and Schools (from Open Data US). Statistics are reported in Appendix Figure C.7
Company Employees, US Elections,
For all datasets, the values of the query columns must. be matched with the entities in YADL using semantic annotation solutions (Huynh al. [022 Nguyen al.) [2021). In our experiments, we manually performed the match to remove the noise from this task. Also, query columns are chosen based what user may reasonably consider as "key" (e.g., the company name). We release the matched tables along with the data lakes for reproduciblity.
To reflect the experience of a data scientist that needs to construct a meaningful table starting from a data lake, and to highlight the effect joins on the downstream task, we run experiments on a depleted version of the tables, i.c., the input tables include only the primary key column and the target column. We provide details on the datasets, computational resources, cross-validation setup, and pre-processing in Appendix C.2
5.2 Results
Our first goal is to pinpoint which steps of the pipeline have the most significant impact on the studied dimensions:
optimizzng these influential sections can yield the greater benefits. We rely on Pareto optimality plots Figure 2 highlight the performefore trade-off for the different steps and configurations.
define reference configuration that uses Exact Matching, Best Single Join, aggregation Any, and CatBoost. This configuration represents good trade-off between performance and compute cost, and it can run on all configurations, base tables and data lakes.
Reference configuration Based on Figure
The data lakes available at httpp://doi. www.oor.g./////.....o. wtw.11/..52122eedoo.oo 106000000000001000000000000 the code prepare YADL https: //github.
and the pipeline htttp:://itthbb. http://aadaiirraarrreeeeeeeeeeee preetpp
8

---

Published in Transactions on Machine Learning Research (5/2025)
Retrieval method
Selector
Predictor
Exact
H. MinHash
Full Join
Step. Greedy Best Single J.
CatBoost
ResNet
MinHash
Starmie
Highest JC
RidgeCV
RealMLP
0.5
0.0
Pareto
frontier
-0.5
2m
10m
60m 250m
2m
10m
60m 250m
2m
10m
60m 250m
0.5
0.0
0.5
2GB 4GB 8GB 15GB 145GB
2GB 4GB 8GB 15GB 145GB
2GB 4GB 8GB 15GB 145GB
Figure 2: Pareto diagram for the pipeline steps The prediction performance (y-axis) is plotted against retrieval + run time (top) and peak RAM usage (bottom). Each row presents the same results, broken down by retrieval method (left), join selector method (center) and predictor (right). Each dot represents the average prediction performance and resource cost. averaged across base tables and data lakes for a specific configuration (e.g., the leftmost dot in the first row is obtaining by using MinHash, Highest Containment Join and CatBoost). Time for offline retrieval preparation for MinHash and Starmic not reported here.
Table 2: Ablation study, median of the prediction difference (higher is better) and runtime difference (lower is better) from the reference configuration. Retrieval Method does not include results from Open Data US and YADL 50k.
Retrieval
Diff. Prediction
Diff. Time
Selector
Diff. Prediction
Diff. Time
Exact Matching
0.00 %
1.00x
Full Join
2.09 %
0.69x
Starmie
-0.06 %
2.20x
Stepwise Greedy Join
1.61 %
2.10x
Hybrid MinHash
-3.19 %
0.67x
Best Single Join
0.00 %
1.00x
MinHash
-17.48 %
0.34x
Highest Containment Join
-0.10 %
0.45x
Aggregation
Diff. Prediction
Diff. Time
Predictor
Diff. Prediction
Diff. Time
DFS
2.46 %
4.42x
CatBoost
0.00 %
1.00x
Mean
0.02 %
1.02x
RealMLP
-21.24 %
17.52x
Any
0.00 %
1.00x
ResNet
-24.11 %
5.92x
RidgeCV
-26.03 %
3.31x
Table 2 reports an ablation study comparing the performance and runtime of the reference configuration against the other possible methods; we report the median difference in prediction and runtime.
Predictor: Treo-based methods perform well Supervised learning with the ML model the pipcline step with the starkest. difference between methods Fiiure right, Table 2): in our scenario, CatBBoost. is both faster and more effective than its non-tree counterparts. Indeed, CatBoost always outperforms other methods in prodiction performance (by up to 26%) and run-time (17x faster than the slowest competitor).
4This is expanded in Appendix subsection C.2 and Figure 14

---

Published in Transactions on Machine Learning Research (5/2025)
This is likely due to two major factors: 1) imperfect joins introduce large fraction of missing values (database nullss") and 2) categorical features have high cardinality. Firstly, any sample that does not find match in a candidate will have a missing value in each new feature; in addition, any missing value in matched rows will remain: as a result, even "good" joins may feature a high degree of missingness. Parametric models are particularly affected by missing values, and categorical features make imputation more dlffficult; comparatively, tree-based models are far more resilient in presence of missing values (Josse et al., 2024). In fact, we observe that CatBoost and RealMLP are the only methods whose prediction performance never drops below 0 even in this very challenging scenario. McElffesh et al. (2023) also report that heavy missingness impedes neural networks more than tree-based models.
Retrieval: Containment is key, though imperfect
A surprising result of our experiments shown in Figure 2 and Table 2 is that simple metric-based retrieval is suf- ficient to achieve good results, while maintaining a far lower resource cost. Relying directly (or indirectly) on Jaccard containment (Highest Containment, Full Join, Exact Matching) achieves the same results as far more
1.0
Data lake
Prediction score
Binary
0.8
YADL Base
YADI 10k
0.6
50k
Open Data
0.4
0.2
thorough (and costly) retrieval (Starmic) or selection (Best
Single Join, Stepwise Greedy Join).
0.25
0.50
0.75
1.00
Focusing on the reference configuration, Figure 3 relates containment to prediction performance. The plotted lines highlight the clear upwards trend in the prediction ion perfor- mance that follows the increase in containment. In other words, higher prediction results occur more frequently when the containment is higher. For exact joins (as in our scenario), having high Jaccard containment leads to better results because left joins with high overlap augment large fraction of the base table, whereas low overlap joins produce features that contain mostly emply values.
Containment
Figure 3: Retrieval: Better containment im- proves prediction performance. Regression plot relating the prediction performance with the Jaccard containment; each dot represents perimental run for each base table.
And yet, while Jaccard containment is an effective first metric, it is not fool proof. We observe experimentally that does not work well where there is large degree of redundancy because such cases many candidates may share the same containment: this is what happens with YADL 50k (Table 3/ Appendix Figure 7): a larger computational budget may mitigate this issue, but it is not always option. In other words, high containment means that we can join without adding too many missing values: this does not guaranlee that the added features improve downstream utility.
Retrieval: Better metrics improve performance, at cost Pareto diagram (Fiuure 2 left) shows that Starmic is Pareto optimal for some (resource-intensive) configurations: if resources available for the offline preparation cost, better querying through Starmie can bring benefits. And yet, the -arguably small- improvements in downstream performance brought by Starmie come with very large cost in both RAM and offline training time Figure 2 Figure 22 Appendix Table 6 Appendix Table 10): Starmie has about 7.5x the RAM footprint of the most expensive alternatives; furthermore, it could only run on out our 5 data lakes of interest (Binary, YADL Base and YADL 10k). On the other hand, Exact matching Parcto optimal in most cases, and runs on all data lakes we considered.
With respect to MinHash, Hybrid MinHash brings substantial improvement in terms of downstream performance Figure 2 Table 2 Figure 14). This comes at the cost of an increase in the query time Figure 18 Appendix 10 which is however amortized due to the cost of performing cross-validation.
The cost of retrieving candidates by testing many query columns is another important factor in the scalability of the retricval methods. Indood, the user does not know which the best query column use, if there are multiple, they may favor methods that have a shorter query time (such as MinHash and Hybrid MinHash) in order to the evaluation pipeline faster. We explore scalability in more detail in Appendix C.1
10

---

Published in Transactions on Machine Learning Research (5/2025)
Value of k
0.54
Prediction performance
0.52
2
Figure 4: Trade-off in prediction perfor- mance the number of candidates in- creases for the Full Join selector. Results are averaged over all base tables and data lakes and represented as a Pareto plot.
0.50
k used in experiments
5
8
0.48
10
0.46
60
0.44
100
0
50
100
Time run (s)
increases To understand the impact of k the prediction performance, record the prediction performance for range of values, and report the results in Figure 4 While there is some variability due to the data lake and base table, we observe that results plateau quickly as l increases, saturating for as low as 8. We explore this facet in more detail in Appendix C.10
Retrieval:
performance plateaus quickly
A Aggregation: Complex aggregation improves per- formance with limitations Figure 5) reports aggre- gation results]] The systematic expansion of multiple aggregations in DFS leads to many features, which would make both Full Join and Stepwise Greedy Join intractable, so they are not considered here. The generated features overall bring useful information, as DFS outperforms the simpler aggregations in most cases; however, preparing and employing the new features increases noticeably the training time. This is confirmed by Table 2 which high- lights both the improvement in performance 2.5%) and the major increase of compute cost. ( (4.42x wrt Any).
Prediction Performance
0.4
0.2
Aggregation
Any
-.2
Mean
-0.4
DFS
2m
10m
60m
250m
Time run (s)
Figure 5: Aggregation: Pareto diagram. DFS outperforms other methods, but is much slower.
Overall results across tables Table 3 reports the
median prediction performance across all configurations for pair "Base table Dala Lake". From this table it is evident that internal tables (Schools, US County Population) represent much simpler problem to solve compared to tables that are not sampled from the data lake (all other tables).
More detail is available in Appendix, Figure 15
Table Aggregated median prediction results (higher better) for the reference configuration. "NA" results are referring to experiments run on "internal" tables ("Schools" was sampled from Open Data, "US County Population" from YADL): if run on a different data lake, no matching entities could be found, new features would be added.
Base Table
Binary
YADL, Base
YADI. 10k
YADL 50k
Open Data US
Company Employees
0.20
0.33
0.37
0.25
0.26
Housing Prices
0.34
0.57
0.54
0.54
0.50
Schools
NA
NA
NA
NA
1.00
2021 US Accidents
0.26
0.44
0.44
0.43
0.31
US County Population
0.93
0.84
0.95
0.85
NA
US Elections
0.44
0.55
0.52
0.52
0.59
11

---

Published in Transactions on Machine Learning Research (5/2025)
6 Discussion and conclusions
Take-away messages We build semi-synthetic data lake based knowledge base to use a reproducible test bed for evaluating methods to augment user-provided tables. We implement an easily- extendable pipeline to test the different steps of the augmentation procedure and alternative algorithms for each of their tasks. Our results uncover a number of observations important to direct research on this subject. We summarize the main take-away messages from the experiments.
1. Tree-based models come with significant benefits in terms of prediction and computational performance in the data-lake pipeline we studied (Figure 2 right, Table 2). Indeod, the automatod selection of augmentation candidate tables generates challenging features (e.g., with many missing values), which tree-based models more effective at dealing with.
2. Good table retrieval affects the entire pipeline by discovering candidates that contain more useful features and introduce fewer missing values (Figure 2 left)- Jaccard containment is good metric for retrieving candidates, but it has limitations (Figure 3/.
3. Simple metric-based retrieval (Exact matching) and candidate selection (Highest containment) produce comparable or better results than more complex methods (Starmie and Best single join), while being
vastly more efficient (Figure 2 left, center, [able 2
4. Performance plateaus quickly the mumber of retrieved candidates increases, while the resource cost increases steadily Figure 4).
5. Complex aggregation methods are much slower than simpler ones and do not result in commensurate gains in prediction performance (Figure 5. Table 2).
6. Combining stochastic retrieval with exact metrics (Hybrid MinHash) mitigates some of the drawbacks of basic MinHash and scales well with many query columns Figure 3
Table 2
The topic that we have studied is at the intersection of database and machine learning research. This intersection still has many open research prospect:
Merging on clean columns is only one side of the story. While we focus on columns where join keys can be matched exactly (functionally, by doing string matching), this is not possible in general because of typos, different formats, and different granularity. Similarity joins and semantic matching would help with
this problem (Jiang et al. 2014] Dong et al., 2023 Deng al. 2017b Mundra el al. 2023) and add another dimension to the analysis. Methods that cover multiple steps, such as |Gonng et al., 2023 Huang et al., 2023c),
could also be considered.
To limit the search space, we keep our chain of joins for augmentation limited to one, and show that scalability is an issue even in this simplified scenario. However, some methods have considered chains of join for augmentation (Galhotra et al. recursive methods (Cveekov- Iliev et al.
2023 EEnabiig join chains would also make cvident the benefit of
2023 Kanter & Veeramachaneni 2015).
" "AutoRetrieval" is an exciting prospect. The overall middling sotellingtt-poorr prediction results (Table 3 Table 8) suggest that the automated strategies evaluated here not sufficient to replicate the performance achieved by a human data analyst. Automated procedures may help with discovering good candidates, but they cannot replace a human expert: fully automating this operation similarly to how AutoMI. tools (Hutter 2019 Feurer et al. 2015) tweak ML methods represents compelling direction for future work.
et al.
Final words. The performance-complestty trade-offs that appear are crucial for the practitioner: they suggest how to automate supervised learning on a data lake of a given size, maximizing statistical performance within compute budget.
Acknowledgements
We acknowledge the invaluable insight and expertise provided by Léo Dreyfus-Schmich and Du Phan (formerly from Dataiku), without whom the foundations of this work would not have existed.
Note that our results are not in contradiction with the Starmie study, focused on table anionability, i.e. finding more samples rather than feature augmentation which our focus.
12

---

Published in Transactions on Machine Learning Research (5/2025)
As YADL is based on YAGO acknowledge the work made by its authors to prepare the original knowledge base, as well their efforts manually evaluating it. This research was supported by the Project P16, ANR-23-RDIA-000L. We express our gratitude for the financial support that made this study possible. Finally, we would like to thank the anonymous reviewers for their constructive comments and valuable suggestions, which have significantly improved the quality of this manuscript.
References
Ziawasch Abedjan, John Morcos, Michael N. Gubanov, Ihab F. llyas, Michael Stonebraker, Paolo Papotti, and Mourad Ouzzani. Dataxformer: Leveraging the web for semantic transformations. In Seventh Biennial Conference on Innovative Data Systems Research, CIDR, 2015. URL Papers/CIDRe CIIR/IIIIR111_/1 1011l__aper..1111 pdf
Scrcan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 6679 6687, 2021.
Jens Bleiholder and Felix Naumann. Data fusion. ACM computing surveys (CSUR), 41(1):1-41, 2009.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE transactions neural networks and learning systemss 2022.
Sonia Castelo, Rémi Rampin, Aécio Santos, Aline Bessa, Fernando Chirigati, and Juliana Freire. Auctus: a dataset search engine for data discovery and augmentation. Proceedings of the VLDB Endowment, 14(12): 2791-2794, 2021.
Nadiaa Chepurko, Ryan Mareus, Emanuol Zgraggen, Raul Castro Fernandez, Tim Kraska, and David R. Karger. ARDA: automatic relational data augmentation for machine learning. Proc. VLDB Endow,, 13(9):1373 1387, 2020. doi: 00.14778/3397230.558
URL http://www. vldb. http:///w./////////////////////e
Tianji Cong, James Gale, Jason Frantz, HV Jagadish, and Çağatay Demiralp. Warpgate: A semantic join discovery system for cloud data warchouse. arXiv preprint orXf::2212.14155, 2022.
Alexis Cvetkov-Iliev, Alexandre Allauzen, and Gaël Varoquaux. Relational data embeddings for feature enrichment with background information. Machine Learning, pp. 1-34, 2023.
Dong Deng, Raul Castro Fernandez, Ziawasch Abedjan, Sibo Wang, Michael Stonebraker, Ahmed K Elma- garmid, Ihab F Ilyas, Samuel Madden, Mourad Ouzzani, and Nan Tang. The data civilizer system. In Cidr, 2017a.
Dong Deng, Albert Kim, Samuel Madden, and Michael Stonebraker. Silkmoth: An efficient method for finding related sets with maximum matching constraints. arXiv preprint arxiv:1 04 738, 2017b.
Xin Luna Dong, Laure Berti- Berti-Equille, and Divesh Srivastava. Truth discovery and copying detection in a dynamic world. Proceedings of the VLDB Endowment, 2(1):562 573, 2009.
Yuyang Dong and Masafumi Oyamada. Table enrichment system for machine learning. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 3267-3271, 2022.
Yuyang Dong, Chuan Xiao, Takuma Nozawa, Masafumi Enomoto, and Masafumi Oyamada. Deepjoin: Joinable table discovery with pre-trained language models. Proc. VLDB Endow. 16(10):2458 22222444--7777 2023. doi: 10.14777/3300551133600
URL https://www. vldb. http.///rg//o.a////oo///////---0 .df
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Podro Larroy, Mu Li, and Alexander Smola. Autoghuon-tabulan Robust and accurate automl for structured data. arXiv preprint arXiv:2003.0650: 2020.
13

---

Published in Transactions on Machine Learning Research (5/2025)
Mahdi Esmailoghli, Arnulfo Quiané-Ruiz and Ziawasch Abedjan. COCOA: correlation coefficient- aware data augmentation. In Proceedings of the 24th International Conference on Extending Dalabase Technology, EDBT, pp. 331 336. OpenProcceddinssorg 2021. doi: https://doi..... oww.orgg//// 10. 5441/002/ 2021.30
2021.30 URL
Grace Fan, Jin Wang, Yuliang Li, and Renée J. Miller. Table discovery in data lakes: State-of-the-art and future directions. In Companion of the 2023 International Conference on Management of Dala, SIGMOD '23, pp. 69-75, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9781450395076. doi: 10. 10.1145/3555044 355504113589444 URL https://dol.. fttps::/ddo.orr/100
1145/3555041. 3589409
Grace Fan, Jin Wang, Yuliang Li, Dan Zhang, and Renée J. Miller Semantics-aware dataset discovery from data lakes with contextualized column-based representation learning. Proc. VLDB Endow., 16(7):1726 1739, 2023b. doi: 40.14778/3588736.
URL https://wwx
widb. www.wllwllllll..e.oooo httppppp.dd/vvll1//p1112------11 pdd
Raul Castro Fernandez, Ziawasch Abedjan, Famien Koko, Gina Yuan, Samuel Madden, and Michael Stone- braker. Aurum: A data discovery system. In 2018 IEEE 34th International Conference Data Engineering (ICDE), pp. 1001 1012. IEEE, 2018.
Raul Castro Fernandez, Jisoo Min, Demitri Nava, and Samuel Madden. Lazo: A cardinality-based method for coupled estimation of jaccard similarity and containment. In 2019 IEEE 35th International Conference on Dala Engineering (ICDE), pp.
IEEE, 2019.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances Nearul Information Processing Systems 28 (2015), pp. 2962-2970, 2015.
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-frce automl via meta-lcarning. Journal of Machine Learning Research, 23(26)):1-61, 2022.
Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. Position: Relational deep learning graph representation learning on relational databases. In Forty-first International Conference on Machine Learning.
Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. Relational deep learning: Graph representation learning relational databases. arXiv preprint ar Xiv:23112 04615, 2023.
Javier Flores, Sergi Nadal, and Oscar Romero. Towards scalable data discovery, 2021. URL https: : //openproceediigg..omm ort.ooog/////// cotf////oo///// wwtp:///tttp.../////////
Sainyam Galhotra, Yue Gong, and Raul Castro Fernandez. Metam: Goal-oriented data discovery. In 39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, CA, USA, April 3-7, 2023, pp. 2780 2793. TEFE, 2023. doi: 10. 10.100///CDF5555552222... 2002230055555022002020233 ICDE55515. 2023. 00213
URL httpp::/doi.org//// 1109/
Yue Gong, Zhiru Zhu, Sainyam Galhotra, and Raul Castro Fernandez. Ver: View discovery in the wild. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 508-516. IEEE, 2023.
Google. Datasct scarch, 2023. URL https:/dattasetsaarc.. aesaaaaaattaarrrc..eeeaaccch weeeearchh.gogge.coo com/
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning mod- cls for tabular data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advonces in Neural Information Processing Systems, volume 34, pp. 18932-18943, Cur- Associates, Inc., 2021. URL wtthhtttss//ppoeeelgs.......rppp htt.e/dddwnnneeerrss.se.eoeeeeee cc/paper http:///w...cccpppppre/////////p BaGG63199FF7744eeeea333222---2-3 pdf
Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning moets nearest neighbors. In The Twelfth International Conference on Learning Representations, 2024.
14

---

Published in Transactions on Machine Learning Research (5/2025)
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Conference on Newral Information Processing Systems, 2022.
Isabelle Guyon and André Elisseeff. An Introduction to Variable and Feature Selection. Journal of Machine Learning Research, 31M7)):1157-1182, 2003. ISSN ISSN 1533-7928.
Maryam Habibi, Johannes Starlinger, and Leser. Tabsim: A siamese neural network for accurate est imattion table similarity, In 2020 IEEE International Conference on Big Data (Big Data), pp. 930 937, 2020.
doi: 10.1109/Biii 20t.Daaaaa00222220..9933333
Alon Y. Halevy, Flip Korn, Natalya Fridman Noy, Christopher Olston, Ncoklis Polyzotis, Sudip Roy, and Steven Euijong Whang. Goods: Organizing google's datasets. In SIGMOD, pp. 795 806. ACM, 2016. doi:
10. 10.1145 ////2/2229032223333. URL https://doi org/10. 1145/2882903 2903730
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art, Knowledge- systems, 212:106622, 2021.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoşi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference Artificial Intelligence and Statistics, pp. 5549-5581. PMLR, 2023.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint (rXim:2207.01844 2022.
David Holzmüllerr, Leo Grinsztaju, and Ingo Steinwart. Better by default: Strong pre-tuned mips and boosted trees on tabular data. arXiv preprint arXim:2407.04441, 2024.
Zezhou Huang, Jiaxiang Liu, Daniel Gbenga Alabi, Raul Castro Fernandez, and Eugene Wu. Saibot: A differentially private data search platform. Proceedings of the VLDB Endowment, 16(11):305733077 2023a.
Zezhou Huang, Jiaxiang Liu, Haonan Wang, and Fugene Wu. The fast and the private: Task-based dataset. search. Xiviv preprint arXic:2308. 05637, 2023b.
Zezhou Huang, Pranav Subramaniam, Raul Castro Fernandez, and Eugene Wu. Kitana: Efficient data augmentation search for automl. arXiv preprint arXiv:2305.10419 2023c.
Madelon Hulsebos, Çagatay Demiralp, and Paul Groth. Gittables: A large-scale corpus of relational tables. Proceedings of the ACM on Management of Data, 1(1):1-17, 2023.
Madelon Hulsebos, Wenjing Lin, Shreya Shankar, and Aditya Parameswaran, It took longer than i was expecting: Why is dataset search still so hard? In Proceedings of the 2024 Workshop Himaa--In-the-Tove Data Analytics, pp. 1 1-4, 2024.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning Methods, Systems, Challenges. Springer, 2019.
Viet-Phi Huynh, Yoan Chabot, Thomas Labbé, Jixiong Liu, and Raphaël Troney. From Heuristics to Language Models: Journey Through the Universe Semantic Table Interpretation with DAGOBAH. In Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab), 2022.
Yu Jiang, Guoliang Li, Jianhua Feng, and Wen-Syan Li. String similarity joins: An experimental evaluation. Proceedings of the VLDB Endowment, 7(8):625 636, 2014.
Julie Josse, Jacob M. Chen, Nicolas Prost, Gaël Varoquaux, and Erwan Scornet. On the consistency of supervised learning with missing values. Statistical Papers, September 2024. ISSN 1613-9798. doi:
10.1007/600362--2--110--
James Max Kanter and Kalyan Vecramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference on data science and advanced analytics (DSAA), pp. 1-10. TEEE, 2015.
15

---

Published in Transactions on Machine Learning Research (5/2025)
Shubhra Kanti Karmaker, Md Mahadi Hassan, Micah J Smith, Lei Xu, Chengxiang Zhai, and Kalyan Veeramachaneni. Automl to date and beyond: Challenges and opportamities. ACM Compuling Surveys (CSUR), 54(8):1-30 36, 2021.
Aamod Khatiwada, Roee Shraga, Wolfgang Gatterbauer, and Renée J. Miller. Integrating data lake tables, Proceedings of the VLDB Endowment, 16(4):932 945, dec 2022. doi: 10.14778/3554225535777777 URL
https://doi.org:: 10/g//1..10 0tt.://.00001111117//33344422230 3574274
Aamod Khatiwada, Grace Fan, Roee Shraga, Zixuan Chen, Wolfgang Gatterbauer, Renée J. Miller, and Mirek Riedewald. SANTOS: relationship-based semantic table union search. Proc. ACM Manag. Data, (1):9:1 9:25, 2023a. doi: 00.1145/3588689 URL
Aamod Khatiwada, Roee Shraga, and Renée Miller. Dialite: discover, align and integrate open data tables. In Companion of the 2023 International Conference on Management of Dala, pp. 187 190, 2023b.
Arun Kumar, Jeffrey Naughton, Jignesh M Patel, and Xiaojin Zhu. To join or not to join? thinking twice about joins before feature selection. In Proceedings of the 2016 International Conference Monagement of Data, pp. 19-34, 2016.
Jiabin Liu, Chengliang Chai, Yuyu Luo, Yin Lou, Jianhua Feng, and Nan Tang. Feature augmentation with reinforcement learning. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, may 2022. doi: 10. 1111109//cee337422222222200
URL
http://..rrg/...11/////eeeeeeee 200.:37774..2220031311 2022.00311
Farzaneh Mahdisoltani, Joanna Biega, and Fabian Suchanek. Yago3: A knowledge base from multilingual wikipedias. In 7th biennial conference on innovative data systems research. CIDR Conference, 2014.
Chris A Mattmann, Sujen Shah, and Brian Wilson. Marvin: An open machine learning corpus and environment for automated machine learning primitive annotation and execution. ar arXie preprint arXiv: 1800:38380,733 2018.
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When Do Neural Nets Outperform Boosted Trees on Tabular Data? Advances in Neural Information Processing Systems, 36:76336-76369, December 2023.
George A. Miller. WordNet: A lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994. URL httpss//acllamthooogy..rrg 1111
Jan Motl and Oliver Schulte. The ctu prague relational learning repository. arXiv preprint arXiv: trXiu:1511.0006 2015.
Pranay Mundra, Jianhao Zhang, Fatemeh Nargesian, and Nikolaus Augsten. Koios: Top-k semantic overlap set search. In 2023 IEEE 39th International Conference on Dala Engineering (ICDE), pp. 1531-1543. IEEE, 2023.
Fatemeh Nargesian, Erkang Zhu, Ken Q. Pu, and Renée J. Miller. Table union search on open data. Proceedings of the VLDB Endowment, 11 (1(7):813-82) mar 2018. doi: 10. 144778/31929653119999 URL
http:///oo..rrg/1..444/////
31929733
Fatemeh Nargesian, Erkang Zhu, Renée J. Miller, Ken Q. Pu, and Patricia C. Arocena. Data lake management: 1980))9986999888 1989, 2019. doi: 1114778/33520333333 16.
Challenges and opportunities. Proc. VLDB Endow., (2(12):1986
URL http://www. vidd. http://wwwgppppolooo//////a/aaaa
Phuc Nguyen, Ikuya Yamada, Natthawut Kertkeidkachong Ryutaro Ichise and Hideaki Takeda. Semtab 2021: Tabular data annotation with mtab tool. In (ISWC, volume 3103 of CEUR Workshop Proceedings, pp. 92-101. CEUR-WS.org, 2021. URL http:///////////oooooo//////////
Andrei Paleyes, Raoul-Gabrit Urma, and Neil D Lawrence. Challenges in deploying machine learning: a survey of case studies. ACM compuling surveys, 55(6):1-29, 2022.
16

---

Published in Transactions on Machine Learning Research (5/2025)
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825 2830, 2011.
Du Phan. Enhancing Data Analytics Through Intelligent Join Suggestions. aaaooossscnaaannnnnnaaaaaaaaaaaa wtw.ee/wwaaa.alyooooooooooooeeee hogpoggl-mmmlllggoojjoo--ngggggo sughossssggggssnnss
January 2023.
Liudmila Ostroumova Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey
Gulin. Catboost: unbiased boosting with categorical features. In NewrIPS, pp. 6639 6649, 2018. URI,https. http://t9--btttatttaeaea
\/proceedings
Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovee. Relbench: benchmark for deep learning relational databases, 2024. URL https: /tttp::/arxii...o www.aawwwwww..oooooooooooeoooeee attp///b//222...0 2000/
Aécio Santos, Aline Bessa, Fernando Chirigati, Christopher Musco, and Juliana Freire. Correlation sket ches for approximate join-correlation queries. In Proceedings of the 2021 International Conference on Management of Dala, pp. 1531 1544, 2021.
David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Francirnc Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. Advances in neural information processing systems, 28, 2015.
Vraj Shah, Arun Kumar, and Xiaojin Zhu. Are key-foreign key joins safe to avoid when learning high- capacity classifiers? Proc. VLDB Endow. 11(3):366 379, 2017. doi: 10.11778/315779933577944 URL
http://www. http:://wwwwlllbb...mmm vldbb.rggg./...............o.... wttgpvvddbvvol////eeeee
shah shah. pdf
Ravid Shwartz Ziv and Amitai Armon Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
Engineering Spotify. httpp://enggineering, htw.:/gneeernng.atttiit...oom hwwp:///wwwwwweeoeeeeeeeeeeeeeee spotify/, February 2020.
How We Improved Data Discovery for Data Scientists at Spotify.
Kavitha Srinivas, Julian Dolby, Ibrahim Abdelaziz, Oktie Hassanzadeh, Harsha Kokel, Aamod Khatiwada, Tejaswini Pedapati, Subhajit Chaudhury, and Horst Samulowitz. Lakebench: Benchmarks for data discovery over data lakes. Xixiv preprint rxXi:2307.04211 2023.
Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pp. 697- 706, 2007.
Ritchie Vink, Stijn de Gooijer, Alexander Beedie, Marco Edward Gorelli, van Zundert, Weijie Guo, Gert Hulselmans, universalmind303,, Orson Peters, Marshall, chielP, nameexhaustion, Matteo Santamaria, Daniël Heres, Josh Magarick, ibENPC, Moritz Wilksch, Jorge Leitao, Jonas Haag, Marc van Heerden, emdlineluser, Oliver Borchert, Chris Pryer, Ryan Russell, Joshua Peek, Colin Jermain, Adrián Gallego Castellanos, Jeremy Goh, and Liam Brannigan. pola-rs/polars: Python polars 0.20.6, January 2024. URL
http::///aoi.rr//0.... http:/10.521//enoddae http://22oodo..177888838 10573881
Jiayi Wang, Chengliang Chai, Nan Tang, Jiabin Liu, and Guoliang Li. Coresets over multiple tables for feature-rich and data-efficient machine learning. Proceedings of the VLDB Endowment, 16(1):64 76, sep 2022. doi: 10.14778/3560 111.3662667 URL httts:://o::::: org/ 10. 117714437717222 3561267
Minjie Wang, Quan Gan, David Wipf, Zhenkun Cai, Ning Li, Jianheng Tang, Yanlin Zhang, Zizhao Zhang, Zunyao Mao, Yakun Song, et al. 4dbinfer: A 4d benchmarking toolbox for graph-centric predictive modeling on relational dbs. arXiv preprint 2rXiv:2404.1822 2024.
Han-Jia Ye, Si- Yang Liu, Hao-Rum Cai, Qi-Le Zhou, and De-Chuan Zhan. A closer look deep learning on tabular data. arXiv preprint or orv:: Xiv:2407.00956, 2024.
17

---

Published in Transactions on Machine Learning Research (5/2025)
Zixuan Zhao and Raul Castro Fernandez. Leva: Boosting machine learning performance with relational embedding data augmentation. In Proceedings of the 2022 International Conference on Management of Data, pp. 1504 1517, 2022.
Eric Zhu, Vadim Markovtsev, aastafiev, Wojeiech Lukasiewiez, Adam Foster, Sinusoida136, Andrii Oriekhov, Joe Halliwell, JonR, Kevin Mann, Keyur Joshi, Peter Kubov, Qin TianHuan, Senad Ibraimoski, Spandan Thakur, Stelano Ortolani, Tilusz, Vojtech Letal, Zac Bentley, fpug, bguhlich, long2ice, oisincar, and ronassa. v1.5.9, February 2023. URL https: sttpp://dol: http::///dooor//11... 5281/22nod..
Erkang Zhu, Fatemeh Nargesian, Ken Q. Pu, and Renée J. Miller. LSH ensemble: Internet-scale domain scarch. Proc. VLDB Endow., 9112):1166-1196 2016. doi: 1014771//29450922945444 URL http:
//www.vldb.o vldb. www.wlllbb..oopoooo http:///w....///////////11/////0
Erkang Zhu, Dong Dong, Fatemeh Nargesian, and Renée J Miller. Josic: Overlap similarity scarch for finding joinable tables in data lakes. In Proceedings of the 2019 International Conference Manogement of Data, pp. 847-864, 2019.
18

---

Published in Transactions on Machine Learning Research (5/2025)
Table 4: "City" seed table built by flattening predicates.
Subject
locatedIn_
locatedIn 2
owns
density
Paris
France
Europe
Tour Eiffel
null
Rome
Italy
null
Olympic Velodrome
2236
Table 5: "City" seed table built by left-joining binary tables.
Subject
LocatedIn
Owns
Density
Paris
France
Tour Fiffel
mill
Paris
Europe
Tour Fiffel
mill
Rome
Italy
Olympic Velodrome
2236
A Construction of YADL
We construct YADL by first reshaping the RDF triplets found in YAGO into binary tables, then, depending the construction strategy, we assemble new YADL variants by joining binary tables each other.
Binary tables A binary table is generated for every predicate in YAGO, e.g., "hasCapital", YAGO contains 70 predicates, which result in a small data lake where some of the tables have millions of rows (e.g.. "isLocatedIn") and some have few (e.g., "hasTLD"). Each table contains an attribute named "subject" and another attribute named as the actual predicate. For example, the triplct "France - hasCapital - Paris" leads to lable "hasCapital" with columns "subject" " and "hasCapital"; finally, a row "France, Paris" added to the table. The same process is applied all 70 relations and their triplets.
forddett-bssed tables To reflect that tables typically have more than two columns, we develop a socond variant of YADL where tables have larger number of columns. To this goal, we leverage the Wordnet (Miller 1994) classes (e.g., "Person", "Company", "Artist") to which entities belong. We use all Wordnet classes (a total of 1015) to create our seed tables, following the example in Table 4 Initially, the seed table includes solely the "Subject" columnn (e.g., the "City" table includes only values "Paris," "Rome," " etc.). Then, we join every relation associated with subjects in the table: in Table 4. the subjects with type "City" are joined with relations "locatedIn", "owns", "hassPpDeensity".
Transforming triplet tables into wide format tables introduces mull values because not all entities of a given type have the same relations. For example, column "hasPopDensity" contains null values for subject "Paris" in Table The fraction of missing values in the transformed tables is reported in Table 1
One-to-many relations A subject may be linked to many objects through the same predicate, e.g., "Paris" is linked to two objects ("France", "Furope") via the predicate "locatedIn". We address the issue in three ways. For Binary, triplets that share the same subject and predicate are transformed into tuples with the same value in the "subject" column and different values in the predicate column, e.g., the binary table has columns "Subject" and "isL.ocatedIn" and contains tuples (Paris, France) and (Paris, Europe). For "Wordnet." seed tables, we build two variants. In the first case (which we dub YADL Base in (Section 5), we flatten the group of objects by creating new columns, thus moving them on the same tuple rather than splitting them (e.g., in Table 4 column "locatedIn" is flattened over "locatedIn 1" and "locatedIn _2 In the second variant (YADL 10k and VADL 50k), we create the new columns by executing a left join between the subject and each binary table. One-to-many matches (such as with "Paris" and "France", and "Paris" and "Europe") are replicated with each left. join; ordinarily, this would lead to an explosion in the number of rows: we limit the issue by selecting only the first two entries for each predicate (as shown in Table 5).
As a result of handling these relations differently in each version of YADL, the aggregation step in the pipeline affects them in different ways, thus exposing different. problems.
19

---

Published in Transactions on Machine Learning Research (5/2025)
Building more sub-tables We create supplementary tables derived from the Wordnet seed tables to augment the table count in the data lake with two different methods. In real-life lakes, numerous versions or variants of the same table are common, encompassing collections of tables that undergo slight modifications over time (Halevy et al.] 2016). This redundancy poses a challenge for information retrieval methods in distinguishing between perlinent and extraneous tables. Hence, it constitutes a crucial aspect in a benchmark data lake like YADI,, which incorporates it heavily in the generation of sub-tables.
For YADL Base, generate all combinations of arity 2 and 3 for each seed table. For Table 4| these combinations would include "(isLocatedIn 1, owns "wmss_)", "(owns 1, popDensity)", and "(isLocatedIn 1, owns 1, popDensity)". Each sub-table is then built by projecting the seed table over the generated combination; we retain rows that contain at least one non-null value and drop sub-tables with fewer than 100 rows. For YADL 10k and 50k, we drop all seed tables whose arity is lower than a minimum arity A; then, for each surviving seed table T, we generate N sub-tables by sampling a random subset of columns of size 1A-2, and projecting onto T; each paramcter can be tweaked to modify the size and number of resulting sub-tables. Row-wise redundancy is provided by randomly sub-sampling a fraction p of each resulting sub-table n, more times, thus replicating fraction of the samples while keeping the set of columns fixed. We construct YADL 10k and YADL 50k by setting N to 10 and 50 respectively, A = 8, p = 0.7, and n, = 2.
B Detailed description of the tested methods
B.1 Candidate retrieval
Exact Matching We compute Exaci Matching (Exact) by measuring the exact Jaccard containment for each pair (query column, candidate column) in the data lake, This can be implemented efficiently by first building a "vocabulary" on the query column, and then scamning the whole data lake to compute containment. Candidates are ranked by highest containment, then we select the "top-k" candidates in the pool. An advantage of this method is that computing the containment is an embarrassingly parallel operation.
The main drawback of Exact Matching is the computation, whose cost depends directly on the size of the data lake and the tables therein; furthermore, the operation must be repeated for every new query column. As we show in Figure 18 the cost of querying multiple columns quickly adds up. Another drawback is that containment is less reliable if the data lake features a lot of redundancy (e.g., YADL 50k in Figure 7], since unrelated tables may still feature high containment: this problem may occur if the data lake includes multiple variants of the same table, a setting commonly occurring in industry (Halevy et al., 20166. An additional drawback of Jaccard containment that it does not consider cardinality, so that binary tables would be considered as perfect matches while being useless in practice.
Minnashh (MinHash) MinHash Zhu et al. 2016 relies on Locality Sensitive Hashing (LSH) to build an index subject to a user-set minimum containment threshold. At query time, all candidate columns with estimated containment larger than the threshold are returned. For consistency with the other methods, we select. k candidates from this pool.
The main drawback of MinHash is that it does not feature an inherent ordering of candidates. Moreover, since MinHash returns an approximate result, it may happen that candidates with an actual containment lower than the threshold are returned as False Positives. Under tight constraints, no ordering and False Positives reduce the likelihood of retrieving good candidates Figuue 16). Finally, while the index can be updated whon new data is added to the data lake, performance may deteriorate after a certain level of updates, especially depending on the skew of the new data.
Hybrid MinHash To address Minhash's lack of a candidate ranking, we propose Hybrid MinHash. Hybrid MinHash leverages the prebuilt index from MinHash to find candidates for a given query column, then uses Exact Matching to rank all candidates. The "top-k" candidates are then selected from the ranked list. By measuring the containment over the MinHash query result, the pool of candidates is reduced compared to testing all columns in the data lake (like with Exact Matching).
20

---

Published in Transactions on Machine Learning Research (5/2025)
Just as it takes some of the strong suits of both MinHash and Exact, Hybrid MinHash also shares some of their disadvantages: 1) The MinHash index will degrade each time it is updated. 2) As the first filtering relies on the MinHash results, False Positives increase cost and any candidate missed by MinHash is lost. 3) Ro-ranking candidates by calculating the exact containment increases the query time substantially with respect to MinHash (Table 10.
Starmie We compare the simple methods described so far against Starmie (Fan et al. 2023b), a SOTA system for table unionability and join discovery. Starmie employs contrastive learning method to train column encoders from pre-trained language models to capture contextual information within tables. Starmie performs candidate retricval by building embeddings for cach column in the data lake and for the query column; it then ranks candidales according to the equation:
Sstarmie
IQ n Cil Sc(ved(Q)
where So is the cosine similarity between the embeddings of the query column Q and candidate Cj.
B.2 Candidate selection
Each selector receives as input a pool of "K" candidates from a given retrieval method on given data lake, the train and test splits of the base table, and the aggregation method to use. The train split is further split into a training (0.8) and validation set (0.2). The model is trained on the final integrated table (i.e., the base table joined with the candidates) produced by the selector of this training split, then the result is scored with the test split.
Highest Containment Join We first consider simple method that assumes that ranking is provided in the previous step. Highest Contaimment Join ranks candidates by exact containment, after measuring it for each candidate. The top-1 candidate is joined; ties are broken by taking one candidate at random. This selector re-ranks candidates by containment even if they have already been ranked by the retrieval method, thus it effectively ignores the previous ranking. Due to its simplicity, Highest Containment Join is very quick to execute, but it likely to select suboptimal candidates.
Best Single Join Best Single Join iterates over each candidate one at time. At each step, it joins the base table on the current candidate, performs the aggregation, trains a ML model, then evaluates the model a held-out validation step. After all candidates have been evaluated, the best candidate and associated ML model are used: the ML model is re-trained over the entire training split and selected for the final evaluation. The Best Single Join is an exhaustive and expensive method.
Full Join The Full Join selector does not include any logic for selecting candidates. Given the base table, the Full Join first aggregates candidates, then joins all of them on the base table. Depending on the budget that is provided and the number of columns in the candidates, this may increase the final number of features in the augmented table by a lot. In most of our experiments, we fix the number of candidates to 30. After joining, the MI. model is trained over the fully augmented table. Since training is performed only once, this method is faster than the exhaustive selection methods (Best Single Join and Stepwise Greedy Join), but the large number of resulting features may increase the memory pressure.
Stepwise Greedy Join Stepwise Greedy Join is an extension of Best Single Join that aims to add more information than what is available in a single table, while reducing the amount of noise added by joining irrelevant tables. Firstly, the baseline performance is measured by training model the base table's training split and validating on the validation split. The base table then becomes the "current table" at the first step of algorithm. Then, candidate selection is performed. Candidates are initially re-ranked by their containment, then each candidate is evaluated sequentially. In each ileration, a candidate is joined on the current table, evaluated on the validation split, then depending on whether it improves performance or not it may be kept or discarded. Any time a candidate retained, its features will be added to the "current table". Candidates that do not improve performance are discarded. The training procodure of Stepwise Greedy Join is expensive, as the training operation is repeated multiple times, and slows down as the number of features
21

---

Published in Transactions on Machine Learning Research (5/2025)
increases. It has however the advantage of avoiding the addition of suboptimal features. Algorithm the steps required.
1
reports
Algorithm Pseudocode of the Stepwise Greedy Join selector.
Require: base table T, candidate set C {ci, C2........0 Cuh, scoring function
Require: chosen candidates set S = 0
Require: predictor ML
T Tirain: Toalid get "re__rrininn_splissss
MT..Gt (Trrin))
score= (MMI..prddee((Tuatta)
ftrain Ttrain
volid = Teatid
C = -rerank(C)
for all in 12 do aggregate(c;)
train Eirain LEFT JOIN oa
valid Evalid LEFT JOIN Ca
ML.fil(U/ravn
score; (M...
if score; > score, then
score, score;
lrann Frasn
te
valid valid
S S U {ci}
else
discard(ci)
t t= T
for all s, E do ==t LEFT JOIN 8; ML.
Single-table selectors and multi-table selectors Highest Containment and Best Single Join produce smaller integrated tables as they only join one candidate, rather than all palential candidates like Full Join and Stepwise Greedy Join: we can therefore classify the two pairs as single-table selectors and multi-table selectors respectivoly. Top-k Full Join can be assigned to either class depending on the value of k.
In all selectors that involve a join, aggregation is carried out prior to executing the join itself. In our pipeline, aggregation is carried out before executing any join by grouping the "right table" by the join key, then applying one of the aggregation functions described above. This to avoid materializing large joins, which have huge cost in memory and time. Due to how aggregation is carried out, the final result is the same before and after the join, executing it before materializing the joined table is more efficient.
All join selectors follow the fit-predict paradigm proposed by scikit-learn (Pedregosa et al. 2011 which simplifies extending the pipeline with new selectors.
B.3 Aggregation
When joining tables for downstream machine learning task, one-to-mann many relationships must be aggregated to avoid replicating samples in the base table which would modify the initial data sampling.
Following the example in Figure 6 a join on "Title" leads to the duplication of the two first rows in the base table, as both title values appear in two rows of the candidate table. How should we aggregate the rows to obtain the best prodiction over the column "Revenue"? We test three different aggregation strategics:
1. Any selects one row from each group, without considering the order.
22

---

Published in Transactions on Machine Learning Research (5/2025)
Title
Actor
Age
Year
M.
Year
Title
Actor
Age
The Martian
2015
630
The Martian
The Martian
Matt Damon
Kristen Wllg
49
52
Left
The Martian
The Martian
Matt Damon
Kristen Wiig
52
49
2015
2015
630
630
Toy Story
1995
394
Toy Story
Tom Hanks
66
Join
Toy Story
Tom Hanks
1995
394
The Shining
1980
47
Toy Story
Tim Allen
69
Toy Story
Shining
NULL
NULL
Aggregation
Title
Actor
Age
Year
The Martian
Matt Damon
50.5
2015
630
Toy Story
Tom Hanks
394
The Shining
NULL
NULL
1980
47
Figure Example of how left. join would duplicate rows from the basc table, and how aggregation keeps the same number of samples as the base table (top left).
YADL Base
YADL Binary Open Data US
YADL 10k
YADL 50k
1.0
0.5
0.0
30
30
30
30
30
100
0
0
100
100
0
100
Figure 7: Exact containment measured over the top 100 join candidates for each base table (represented by a colored line) on each data lake. The x-axis reports the rank, rank 30 the retrieval cutoff we use our experiments.
2. Mean replaces all numerical (categorical) values by the mean (most frequent) of all values for that attribute.
3. DFS (Deep Feature Synthesis Kanter & Veeramachaneni 2015 greedily adds new features by performing different aggregations (mean, median, count, standard deviation, etc.) over each column in the table.
C Experimental results
C.1 Choosing which baselines to use
Including recent publications often turned out be very difficult, the code was either unavailable, or not suited to extensive experiments including Starmie (aan et al.) 2023b), and (partially) Lazo (Fernandez al. 2019). We were able to test Starmic on some of the data lakes (Binary, YADL Base, YADL 10k) though we had to adapt and optimize certain sections of the discovery code that did not scale well to our environment. We were unable to run Starmie on the larger YADL 50k, and the less structured Open Data US. Starmie results are illustrated Section 5) The modified Starmic repository is available here: https://gitthb. com/rcap107/starmiie and the improvement was merged in the main repository. These modifications are code optimizations and fixes for corner cases; they do not change the logic of Starmie.
data lake of the size that investigate in our work. We worked on
23

---

Published in Transactions on Machine Learning Research (5/2025)
Table 6: Recorded peak RAM usage for building and querying the different retrieval methods, All sizes are reported in MB. Runs that failed are marked as "NA"
Data Lake
Exact Matching
Minhash Hybrid
Case
Minhash
Starmie
Version
Binary
0.0
1.4
1.4
501.1
Open Dala US
12.7
434
434
NA
Disk
YADL. 10k
21.3
515
515
522.3
YADL. 50k
28.2
2480
2480
NA
YADL base
21.3
462
462
522.4 4
Binary
1196
226
226
19424
Open Data US
5466
3288
3286
NA
RAM Build
YADL 10k
688
3312
3290
87716
YADL 50k
1259
13697
13697
NA
YADL base
1061
2810
2810
38683
Binary
255
216
216
4567
Open Data US
7789
10472
NA
RAM Query
YADL 10k
388
9138
12679
134869
YADL 50k
781
37332
37327
YADL base
412
6784
85/11
1454377
Lazo relies client-server architecture that requires Elasticscarch v7. We were able to set up the backend environment and integrate the Lazo client in our pipeline, but we are unable to execule querying operations on our data lakes because the client database crashes due to out of memory errors. The wrappers we developed for testing Lazo are available in the main repository retrieeee-err--preddic))
Additionally, for the retrieval step we considered ALITE (Khatiwada al. 2022), KOIOS (Mundra al.) 2023). JOSTE (Zhu et al., 2019), and Saibot (Huang et al. 2023a). Each method requires substantial changes to the original codebase or very specific build configurations to datasets and environments different from those included in the original paper.
For the prediction pipeline, we consider ARDA (Chepurko et al. 2020) and Metam (Galhotra et al. 2023 focusing on Metam as it builds upon ARDA. Similarly to the retrieval case, Metam required substantial changes to be executed on data other than that in the paper and when tested in our environment did not produce augmented tables. We implement some of the solutions suggested by Metam in our Stepwise Greedy Join selector, and we re-use the "Open Data US" data lake and "schools" base table in our experimental section. The modifications made to the Metam code to interact with our pipeline available at https://github wwtp:lcoooeeeeeeeeeeeeeeeeeeeeee wwt.///pp1o//ettaaaaaaaaaaaaaaaa
We were not able to find an open repository for SilkMoth (Deng et al. 2017b), DeepJoin Dong el al. 2023
Mileena Huang et al. 2023b or Kitana (Huang et al. 2023c
C.2 Experimental setup
We run our experimental campaign on a SLURM cluster, fixing the number of threads 32. Nodes have at least 256GB of RAM. Experiments that involved NNs were run nodes equipped with GPUs. Overall, preparing the retrieval methods required about 1050 CPU hours, while the experimental results required about 189k compute hours (about 21 years of equivalent CPU time); CatBoost and RidgeCV were run on CPU, ResNet and RealMLP on GPU. Table reports the equivalent CPU (GPU) time across all experiments, broken by ML model; running experiments with 32 CPUs would reduce this runtime by about 32x.
24

---

Published in Transactions on Machine Learning Research (5/2025)
Outer cross-valin validaalia fold
Test split (20%)
Train split (80%)
Candidate
Selector.fit
Inner train (80%)
Inner valid. split (20%)
Inner train split
Candidate
Inner valid. split
Candidate
ML model fit
ML model prodict
Selector.predict
Candidate selection
Test split
Candidate
Best Candidate
Train split
ML model predict
ML model fit
Predictions
Figure 8: Schema of the cross-valien ross-validation setup used in the training pipeline.
Inpul table (afler join)
Categorical features
Numerical features
30 unique
30 unique values
StandarcSccaler
values
OneilotEncoder
OneHHotEncoder
PCA
Simplelmputer
Simpleimputer
Simplelmputer
Training features
Figure 9: Pre-processing steps applied to input tables before being fed to the parametric ML algorithms. Catboost does not require preprocessing. The names reported here reflect the corresponding seikit-learn (Podregosa et al. 2011 objects.
25

---

Published in Transactions on Machine Learning Research (5/2025)
Implementation details The implementation of the pipeline is in Pythor Exact Matching, aggregation and join operations are implemented using Polars (Vink et al.) 2024) as backend. For the Prediction step, RidgeCV is implemented with Scikit- learn (Pedregosa et al. 2011), CatBoost using the official library Prokhorenkova et al. 2018), RosNet and RealMLP use pytabkit Holzmüller al. 2024].
Table 7: Total equivalent compute hours, days, months and years re- quired to run all the experiments. Single-CPU or single-GPU equivalent time: having CP computer di- vides the time by 32.
We rely the Python implementation of MinHash provided by the Datasketch package (Zhu et al., 2023). For Starmic, we optimize the original implementation so it can scale to work in our environment. We use the official DFS package. Data structures are stored by persisting on disk the ensembles for MinHash, the candidate ranking for each query column for Exact Matching, and the model checkpoints for Starmie; Hybrid MinHash relies the MinHash data structures to work, no additional storage is required for it.
Predictor
Platform
Total compute time
RidgeCV CatRoost
CPU
4y 3m 10d 7h
CPU
ly 3m 29d 21h
ResNet
GPU
5y 6m 23d Oh
ReaLMLP
GPU
10y 7m 23d 3h
Total
Both
21y 9m 26d 8h
Table reports the size on disk of the data structures used for each method.
Retrieval We use a containment threshold of 0.2 for the preparation of the MinHash index, and clamp the number of candidates returned by each retrieval method to 30 (except when specified otherwise). These values were chosen to balance execution time and expected number of candidates given the distribution of containment encountered in the different data lakes Figure 7). For Starmic, use the default paramcters defined in the original repository.
Selection We fix the number of Stepwise Greedy Join iterations to 30: this number is consistent with the number of candidates that are provided in the retrieval stop. None of the other join selectors have parameters to tweak.
Prediction We fix the number of CotBo iterations to 300; we stop training the model 10 iterations after the optimal metric has been detected; set the L2 regularization coefficient to 0.01. As we are interested in evaluating the effect of each pipeline task rather than optimizing the prediction performance, we do not. apply HPO to CotBoost
We use the default paramcters for RidgeCV as used in the scikit-learn implementation.
For RealMTP and ResNet we use the parameters that are set in the pytabkit package as they have been shown in Holzmüller et al. (2024) be the "better defaults".
C.3 Critical difference plot
We use critical difference plots to represent an overall ranking of the downstream prediction performance of all the configurations we considered. Specifically, the prediction metric is averaged over all base tables and data lakes, and the resulting value is used to rank each configuration.
Similarly to Figure 2 (and other Pareto plots), we present the same results after splitting them in retrieval method Figure 10), join selector Figure 11), and ML. model (Figure 12.
C.4 Aggregated results
Figure 13 is prepared like Figure 2, however this version includes results from all data lakes and does not. include Starmie as it could not run on Open Data and YADL 50k. Results are quite consistent with Figure 2 this means that the new data lakes (YADL 50k and Open Data US) do not alter the overall trends much.
In Figure 14
we report the fold vs fold difference in prediction score (N²2 and AUC) and relative execution time with respect to the reference configuration (Exact Matching, Best Single Join, aggregation Any and
Reepssioor:: httpe:///tttaaaeeec/rr//////r/es
26

---

Published in Transactions on Machine Learning Research (5/2025)
She mor Catlen
BB)
Figure 10: Critical difference plot for the retrieval methods: Exact Matching, MinHash, Starmie, Hybrid MinHash.
CaNo R CEN
Booet
Wn
Best
Figure 11: Critical difference plot for the join selectors: Full Join, Stepwise Greedy Join, Highest Containment
Join, Best Single Join.
THES Du Ie CaRoO CGO CARE
coly
Figure 12: Critical difference plot for the predictors: CatBoost, RidgeCV, RealMLP, ResNet.
27

---

Published in Transactions on Machine Learning Research (5/2025)
Table 8: Aggregated prediction results broken by base table and data lake. Starmie results are not reported here because it could run on Open Data US and YADL 50k. As many ResNet and RidgeCV runs did not converge, we report trimmed mean with cutoff 0.2.
Base Table
Binary
YADL Base
YADL 10k
YADL 50k
Open Data US
Company Employees
0.086 0.036
0.081 + 0.043
0.071 + 0.052
0.113 + 0.048
0.011 + 0.019
Honsing Prices
0.16 + 0.068
0.187 + 0.077
0.143 + 0.074
0.199 + 0.092
0.221 + 0.09
Schools
NA
NA 0.178 + 0.054
NA 0.199 = 0.076
NA 0.150 0.058
0.892 + 0.137
2021 US Accidents
0.138 0.065
0.229 + 0.028
US County Population
0.725 + 0.203
0.078 + 0.064
0.075 + 0.074
0.112 + 0.148
NA
2020 US Elections
0.374 0.062
0.373 + 0.115
0.358 + 0.054
0.394 + 0.045
0.400 + 0.06
CatBoost) for retrieval FFiuure 14 (a)), selection Figure 14p). aggregation Figure 14p), and prediction Figuue 141). Individual folds are reported as dots; color palettes depend on the data lake. This is an alternative way of representing the data in Table 2 Note that as we are comparing folds against folds, the difference is when a particular parameter is the same as the reference. We choose Best Single Join as a reference for the selector because DFS could not run with Full Join and Stepwise Greedy Join.
Retrieval Figure 14 aa) shows that Exact Matching and Starmie have very similar performance (a median difference of 0.06%), with Starmie outperforming Exact Matching in some instances (this can also be observed in Figuree 2 and Figure 10); which makes sense as the similarity metric used by Starmic combines Jaccard similarity with the cosine similarity of column embeddings. Base MinHash performs very poorly, with a median difference of -7.44% with respect to Exact Matching. This is not surprising and is likely caused by the lack of candidate ranking combined with the presence of a candidate budget. Hybrid MinHash shows marked improvement over base MinHash, gaining about 14% in median: this confirms that it is an effective strategy to address some of the shortcomings of the base method.
MinHash is faster than the others, despite the fact that the candidate budget 30 for all methods. This is because, on average, the candidates retrieved by MinHash have a much lower containment than those proposed by the other methods (Figure 3d): due to the smaller amount of data to move and use for training
Retrieval method
Selector
Predictor
Exact
H. MinHash
Full Join Highest
Step. Greedy Best Single J.
CatBoost RidgeCV
ResNet
MinHash
ReaLMLP
0.5
0.0
-0.5
2m
10m
60m
250m
2m
10m
60m
250m
2m
10m
250m 60m
0.5
0.0
-0.5
5GB
8GB 10GB
15GB
5GB
8GB 10GB
15GB
5GB
8GB 10GB
15GB
Figure 13: Pareto frontier analysis for the pipeline steps on all data lakes. The prediction performance (y-axis) is plotted against retrieval run time (top) and peak RAM usage (bottom). Time for offline retrieval preparation for MinHlash is not reported here. This figure includes all data lakes, but results with Starmie.
28

---

Published in Transactions on Machine Learning Research (5/2025)
a. Retrieval method
Exact
+0.00%
+100x
Starmie
0.06%
+2.20x
H. MinHashh
3.19%
+0.67X
MinHash
17.48%
+0.34x
b. Join selection method
Join
+2.09%
+0.69x
Step. Greedy
+1.61%
+2.10x
Best Single
+0.00%
+1.00x
Highest J.
-0.00%
+0.45x
20%
0%
20%
0.5x
1x
1.5x
2x
Aggregation method
DFS
+2.46%
+4.42x
Any
Mean
+0.02%
+1.02x
60%
40%
20%
0%
20%
40%
60%
1.5x
d. Predictor
CatBoost
+0.00%
+1.00k
RealMLP
21.24%
17..52x
ResNet
24.11%
+5.92x
RidgeCV
26.03%
+3.31x
80%%
60%
-00%
-20%
0%
1x
1.5x
2x
10x
20x
Company Employeas
Housing Prices
Schools
US Accidents 2021
US Countyy Population
US Elections
Birry
Open Data
YADI Base
VADL 10k
50k
Figure 14: Experimental results across all data lakes, comparing the performance difference between evaluated mothods in different steps of the pipeline. The median difference is reported on the right each plot. (a) compares join selectors, (b) aggregation methods, and (c) ML models. In all cases, results are relative to the "average method".
29

---

Published in Transactions on Machine Learning Research (5/2025)
the models, the runtime is shorter; furthermore, imperfect recall affects the performance of both MinHash and its hybrid variant.
While Hybrid MinHash appears to be faster than Exact Matching in the pipeline, it is important to note that re-ranking candidates incurs a non-nggligible additional cost (Figure 14(a), Table 10).
Overall, the two methods based on precise ranking (Exact Matching and Starmie) outperform the methods based in approximate matching (MinHash and Hybrid MinHash),, suggesting that the computational cost. incurred in measuring exact containment does result in better prediction performance.
Selection The choice of selector shows a clear effect when moving from single-table selectors (Highest. Containment and Best Single Join) to multi-table selectors (Stepwise Greedy and Full Join), bringing a benefit of up to 2.2% in median Figure 14
Stepwise Greedy Join much slower compared to all other methods. This is not surprising, since this selector executes a join and trains a model in each iteration, then re-trains the model at the end of the fit step. In practice, while this added complexity brings an improvement of about 1.6% with respect to Best Single Join, this comes at the cost of 2.1x increase in run time; even worse, this is not enough beat Full Join in all cases, as the latter method is both faster and slightly better in prediction performance. Highest Containment is siguificantly faster than all other solutions because it only re-ranks candidates before joining the top-1 candidate.
The difference in performance between the two single-table selectors (Best Single Join and Highest Contain- ment) is likely due to the fact that, while Jaccard Containment an indicator of a potentially good join, it not sufficient. for selecting the best candidate. For example, multiple candidates may have the same containment value, leading ties. Given the lack of better information, Highest Containment breaks ties random, thus it may select tables that not relevant. Redundant data lakes like YADL 50k (Figure 7] are more alfected by this problem. The significant difference between single- and multi-table selectors is explained by the learning model benefiting from an increased set of features: merging more than one table inherently results in a richer feature set.
It is important to mention that the picture is more complex than what can be observed exclusively from Figure 14, In fact, Figures 19 and 20 show that, while Full Join appears to be better on averago, there are configurations of hyperparameters which it performs worse than Stepwise Greedy Join due to the addition of unrelated features.
A final important observation is that all selectors rely on the candidates proposed by retricval method: if these candidates have poor quality, the selectors cannot compensate for that.
Aggregation Aggregation experiments do not include Full Join and Stepwise Greedy Join because DFS ran out of resources with those selectors. In fact, while DFS brings some benefit in its generation of new features (up to 2.46% in median wrt Any), it is also extremely slow (up to 4.42x wrt the simpler method), and this was in a simpler case where only one table was joined at a time. With multi-table selectors, this problem became even more noticeable and prevented us from testing the multi-table selectors. This very large memory cost is consistent with Cvetkov-lliev et al. (2023).
For what concerns the simpler methods, Mean is slightly better than Any, and has a slightly longer runtime; we did not observe major differences between the two methods in practice.
Predictor The performance difference between CatBoost and all other methods already highlighted in Figure and Figure 12 is confirmed in Figure d). which shows CatBoost outperforming the second best model in RealMLP by about 21.2%, and the worse NN model in RidgeCV by 26%. Interestingly, CalBoost is also faster than all other methods by a significant amount, likely owing to the optimized preprocessing of categorical variables. RealMLP is the better performing NN model, and has shown a good resilience in the very challenging environment that we are considering; however, it much slower than all other methods, even when run on GPUs, requiring up to 17x as much time as Boost.
30

---

Published in Transactions on Machine Learning Research (5/2025)
Retrieval method
Aggregation
Predictor
Selector Highest Single
Exact MinHash
MinHash
Any - DFS
CalBoost
ResNet ReaIMLP
Mean
RidgeCV
0.50
0.25
0.00
-0.50
1p
11
2m
10m
60m
250m
2m
10m
60m
250m
2m
10m
60m
250m
2m
10m
60m
250m
0.25
0.00
-0.25
-0.50
3GB 5GB 7GB
12GB 15GB
3GB5GB 7GB
12GB 15GB
3GB 5GB 7GB
12GB 15GB
3GB 5GB 7GB
12GB 15GB
Figure 15: Pareto frontier analysis testing different aggregation methods. The prediction time (top) and peak RAM usage (bottom). Each row
performance (y-axis) is plotted against retrieval presents the same results, broken down by retrieval method (left), join selector method (center) and predictor (right). Starmie, Full Join and Stepwise Greedy Join are not reported here.
Different Lakes Table reports the trimmed average of the prediction performance across all configurations, broken by base table and data lake. We observe experimentally that many RidgeCV and ResNet runs do not converge, leading to very long run times and extremely large negative values for R². For this reason, we use trimmed mean with cutoff of 0.2. Indeed, the results are so poor because the mean across predictors is lowered substantially by the performance of the parametric models. Internal tables could were only run the data lake they were sampled from, and other cells are reported "NA"
Despite its lower containment compared to YADL-based data lakes Figure 3 and Figure 7), Open Data US shows good results, outperforming the YADL variants in some cases: this may be due to the smaller fraction of mulls, and larger tables on average compared to YADL.
Prediction performance was particularly good on US demographic-adjacent tables, and on the internal datasot "Schools", which achieved perfect classification performance in most cases also visible in Figure 30, where highest containment leads to best performance.
YADL's internal table (US County Population) exhibits a similar, though not quite extreme, behavior to that of Schools. This can be explained by the fact that internal (ables, i.e,, tables that are sampled from the data lake itself, tend to have copies in the data lake, which are likely to contain information that is correlated with the prediction task. This highlights the specific behavior of internal tables, which are routinely used in the experimental campaigns in the literature.
The different prediction performance obtained with each YADL variant suggest that the structure of each data lake impacts the actual prediction performance.
C.5
Smart aggregation brings some benefit at a major cost
While containment impacts heavily both prediction performance and execution time, aggregation has a similar impact on the execution time, without the same degree of improvement for the prediction performance Figure 14c).
Aggregating values always leads to loss of information: in exchange for larger cost, complex aggregation methods that preserve information (DFS) that replace values with better representatives of group
31

---

Published in Transactions on Machine Learning Research (5/2025)
Binary VADL Base
Open Data
YADL 50%
Exact
MinHash
H. MinHash
Starmie
0.0
0.4
0.6
0.8 1.0
Containment
Distribution of containment in the top-200 candidates obtained by each retrieval method on different data lakes.
Figure 16:
(mean) should lead to better prediction performance. To an extent, this is what observe: DFS does improve the downstream performance by decent amount, however this is at major computational cost. This problem is exacerbated by the fact that aggregation must be performed whenever join is executed at any point in the pipeline, including joins executed during join selection. The result is a compounding slow-down of the entire pipeline.
The similar performance of "any" and "mean" can be explained by the execution of joins on key-like columns so that values in these columns tend to be mostly distinct, thus producing the same aggregated value for both "mean" and "any". The slight difference in performance and runtime between the two methods may be attributed to the large fraction of missing values in some of the data lakes: the "mean" method is biased towards selecting the most frequent value, which may be a null. When that happens, more values than needed become nulls, reducing the amount of features and introducing a slight speedup in the overall training.
The relatively unsatisfying performance of DFS is also explained by the fact that we are not fully exploiting capabilities. We consider only join depth-1 chains: at each aggregation step, we join the base table with an additional table, rather than leveraging the recursive generation of features provided DFS. As a result, DFS is not as effective at gencrating features as it would be with deeper join paths "Cvekkov-liev et al. 2023 and may benefit from integrating join path discovery systems (Deng al. 2017a).
explaimes
Figure 15 confirms that, while DFS improves the prediction performance, its overall cost remains problematic.
C.6 Distribution of the containment for each retrieval method
To understand the success of the different retrieval methods, we look at the containment of the joins that they suggest. Figure 16 reports top-200 containment. It highlights that, even when retricving a large number (200) of candidates, the average containment MinHash is very low compared to the other mothods: for larger datasets, MinHash returns thousands of candidates of which only a fraction are selected. The lack of an internal ordering means that high-containment candidates are likely be missed (Section 4p. While MinHash has threshold that can be tweaked to reduce the number of candidates that are retrieved, we observe that recall drops sharply at high thresholds. Hybrid Minhash has some success in miligating the problem, as it improves the average containment and the overall downstream performance (Fiuree 2 left). B We were unable to obtain the containment results for Starmie on YADL 50k and Open Data US.
8The performance of MinHash Binary artefact: the method could only retrieve about candidates average compared the found by the other two methods, thus values are averaged fewer candidates with higher containment.
32

---

Published in Transactions on Machine Learning Research (5/2025)
Table 9: Statistics for the base tables.
Base Table
Target
# Num. Att.
# Cat. Att.
# Rows
Company
Number of
2
7
3109
Employees
Employees
Housing
House
3
22250
Prices
Price
Movie
Movies
8
7397
Revenue
School
Schools
4
1774
Class
2021 US
Accidents
1
14850
Accidents
by County
US County
County
1
3059
Population
Population
2020 US
Vote %
3
22093
Election Results
by County
C.7 Execution time breakdown
Figure 17 breaks down where the time is spent by each join selector, with the corresponding average total time the right. Prepare tracks the time spent. building data structures (including any ranking operation), and loading data. Train(model) and Predict(model) track the time spent inside the ML model for training and prediction, re- spectively. Finally, Triin(join) and Predict(join) track the time spent executing a merge operation, combining join and aggregation. Results are aggregated over all experi- ments with "first" as aggregation. The time distribution follows expectations: most of the time is spent fitting models, with time spent joining (and aggregating) in sec- ond place. Highest Containment Join spends a relatively long time in the "Prepare" step due to the need to re-rank candidates before joining: since the join and train steps involve only one table, they are faster in comparison. Step- wise Greedy Join has a similar re-ranking step, however training the models in each itcration dominates the other steps. Full Join merges all candidates at the same time, then trains single model on the result: this explains how the fraction of time spent joining is larger than in other methods. Top-1 Full Join is reference of how long it would take to simply join one candidate and train on that, without any additional operation: as expected, it very fast.
Predict(join) Train(join)
Predict(model) Train(model)
Prepare
Join
Top-1 Join Highest JC Join
1.63
6.70
Join
26.82
Best Single Join
31.85
Step. Greedy Join
0.5
1.0
Frac. spent in sections
Figure 17: Breakdown of where time is spent for each selector. On the right we report the average run time of each selector.
C.8 Computational trade-offs
Figure 22 compares resources for the different retrieval methods indering and query time Starmie by far the most expensive method, both in run time and in peak RAM. The rapid growth of RAA usage, even for relatively small data lakes (see Appendix Table 6), is particularly problematic and prevents the method from running GPUs, thus slowing it down even further. The other methods are far cheaper by comparison, and can run CPU without issues.
Figure 18 shows how different retrieval methods scale with the number of query columns: the value for 0 columns shows the time required to build the index (averaged over all data laked mo and the slope is the
We separate retrieval proper from the pipeline execution clearly highlight their corresponding cost and trade-offs. 10 Open Data and YADI. 50k not included.
33

---

Published in Transactions on Machine Learning Research (5/2025)
time required to query single column (averaged over all query columns); values reported in Appendix Table TO. The figure highlights the point at which the cost of recomputing the containment for each new query column on every column in the data lake (Exact Matching) becomes more expensive than building the index (MinHash), and then querying it.
Tt. clear that, as the number of query columns increases, Exact Matching scales much worse than the other solutions (except Starmie), while Hybrid MinHash and MinHash scale much better: this is important if the query column is unknown
would like to test multiple columns to augment one or more tables. In any case, Starmie is much slower than all other alternatives.
Figure 18 is prepared assuming that query retrieval time increases linearly with the size of a table and that the cost of creating the MinHash index fixed for the data lake at hand; we use Table 10 as reference to build the figure. We observe that the MinHash indexing cost pays itself off after few three queries average thanks to the fast query time; Hybrid MinHlahh requires more queries to break even due to its slower query time, yet it remains faster than Exact Matching when the number of query columns is larger. While the assumptions may not hold in general, the plot gives a reasonable estimate of the break-even points. Starmie scales much worse than other methods it has much slower build and query time, Table 6).
Retrieval time
6h
5h
2h
1h
0h
0
6 8
10
Number of query columns
Figure 18: Runtime as a function of the number of query columns for the different. retrieval methods.
and much larger memory footprint (Fiuure 22
Although dependent on implementation, another factor that should be considered is the size on disk the indices:
the MinHash index occupies a much larger space on disk compared to the data required to hold the Exact matching ranking. Starmie needs to store model checkpoints, which occupy roughly 500MB regardless of the size of the base data lake (Table 6p.
Exhaustive computation of the containment is net gain in performance at the expense of execution time that increases quickly as the number of columns to query increases. This may not be a problem if the user is aware of which columns should be queried; if, instead, the user is trying to conduct an exhaustive search over all columns, a method such as MinHash should be favored. These observations are consistent with Zhu et al. (2019). Finally, scenarios where the query lable and the data lake do not change, query results can computed offline and reused; these scenarios, the additional cost of Exact Matching would be less problematic. In situations where the data lake tends evolve over time, methods that support updating the index such as Fernandez et al. (2019) or Fernandez et al. 20188) should be considered.
C.9 Interplay between different variables
In Figures 20 and we report the interplay between each variable (Predictor, Retrieval method, Selector, Data lake, Base table) with each other. Each row in a figure is boxplot that represents a specific variable, while each column the row breaks down the results by different variable. We provide these figures to showcase how spe- cific configurations may have behaviors that are hidden by the necessity to aggregate information for the main body. Some interesting takeaways that emerge from this representation are the following:
Table 10: Average time required to prepare retrieval methods. Build and query time are separate index construction can be done offline.
Method
Avg. Index time
Avg. Query Time
Exact
826.4
MinHash
2107
2.92
MinHash
2107
473.0
Starmie
18196
512.4
R Holzmüller et al. (2024) is more resilient than all other parametric methods even in the challenging configurations are considering.
RidgeCV is even more penalized by larger tables (see the results with Full Join and Open Data)
34

---

Published in Transactions on Machine Learning Research (5/2025)
CatBoost (CB)
Predictor
ResNet
(RMLP)
RidgeCV (RCV)
Exact (EM)
Retrievva methed
Minilash (HM)))
(SU)
11 0000
MinHash (MH)
Highest JcJ. IHC]
Full Join (F))
TO Dm
Single
Step. Greedy (SWG)
YADL Base (Y-)
(V-10)
Elections (US-E)
Housing Prices (HP)
US County Population (C-P)
US Accidents (US-A)
Employees
1.0
1.0
Predictor
Retrieval methad
Selector
Data lake
Base table
RMLP
St
B5J
[K]
Y10
US-A
US-E
RCV
RN
MH
HMH
SWG
Bin
Figure 19: Effect different variables on the (absolute) prediction performance. Each row reports one variable (o.g., Predictor) and breaks it down by another variable on each column (c.g., by Retrieval method). This plot does not include YADL50k and Open Data US.
Full Join and Stepwise Greedy Join have similar performance, but Full Join has worse worst-case perfor- mance on some data lakes. This confirms the fact. that when noise more prevalent, Full Joining not a good strategy.
Overall, Starmie does not bring a lot of benelits and behaves similarly to Exact Matching in most configurations.
Other than Catboost for predictors, no other method appears to clearly dominate its category.
35

---

Published in Transactions on Machine Learning Research (5/2025)
CatDoost
Predictor
Besbett
RidgeCV
Exact
Retrieval method
MinHash
GBD
Highest
Full Join
Step. Greedy
Best Single
YADL Base
Binary
YADL
Data
YADL
US Elections
Housing Prices
Schools
US County Population
Employnes
Predictor
Retrieval method
Selector
Data lake
Base table
RMLP
SWG
HCJ
RCV
RN
HMH
BS)
Y30
Bin
HP
Figure Elffett different variables on the (absolute) prediction performance. Fach row reports one variable (e.g,, Predictor) and breaks it down by another variable on each column (e.g., by Retrieval method). This plot does not include Starmie.
36

---

Published in Transactions on Machine Learning Research (5/2025)
C.10
Effect of top-k
In this section we give provide additional context on the impact of l on the pipeline. Figure 21 shows that the prediction performance saturates quickly as the number of candidates increases, while the memory footprint increases sharply: this bohavior similar to the impact on runtime shown in Figure 4
Figures 23 24 and 25 show how increasing the value of k affects the prediction performance, fold runtime, function of the data lake and base table. We use the reference the pipeline, with Open Data
and peak R.A.M usage respectively
configuration to obtain these results. The data lake has large impact and YADL1Ok having much larger RAM footprint. This may partially be artefact of how YADL10k is generated; in Open Data's s case, this is likely due to the fact that some of the tables in the data lake contain potentially hundreds of columns.
Value
0.54
Prediction performance
1
Peak RAM
Time difference
0.52
3
Exact
+1.00x
1.00x
0.50
8
MinHash
+15:13x
-167x
0.48
30
0.46
Stamic
283.80X
16.05x
0.44
80
100
1x2x 5x 20x 100x
1x
5xx
20x
2000
3000
4000
5000
6000
Peak RAM fit (MB)
Figure 22: Resource cost of indexing the retrieval methods. The computational cost of each retrieval method is plotted with respect the reference (Exact Matching). The median difference is reported on the right of each plot.
Figure 21: Pareto plot showing how the perfor- mance changes function of the peak fit RAM for different values of k.
37

---

Published in Transactions on Machine Learning Research (5/2025)
Housing Prices
US County Population
Schools
US Elections
US Accidents
Emp mmeloyees
Data
WIDI
VILL STL
Disary
YAOI
1,3
Prent battomance 1r0
0g
0.4
22
40
80 condizates
1.0
20
40
60
100
20
1.00
200
400
80
100
candidates
candidates
Number restleved Candidates
Figure 23: Prediction performance as function of the number of candidates obtained in the retrieval step, broken by data lake.
Sta
55a
SEs
255
255
255
255
255
Smin
125%
321
1.10
325 8:10
125
125 8:0
2mi
2mm
32
40
D2
100
30
40
03
83
1.00
20
10
20
30
100
20
43
62
60
110
23
43
60
10
310
Number restiovod candidates
Numbor retrioedd candidatess
Numbor ofreiiioee canditaaics
Numbor rotfleved canditakess
Numser rotlieved candilanes
Figure 24: Fold runtime in seconds as a function of the number of candidates obtained in the retrieval step, broken by data lake.
Open
12:63
2GB
Peak www
120
122 810
123 B
122
10
123 310
2:03
200
20
300
20
40
50
320
Shurrber
Numtber
TME merdd OWndd dalees
Pourbber
Nummerr
Figure 25: Peak RAM required at fit time function of the number of candidates obtained in the retrieval step, broken by data lake.
38


Reviews:
[{"review_id": "h5u1niWHMW", "reviewer": "Reviewer_H1Bb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper’s primary strength lies in its end-to-end treatment of the data augmentation problem in data lakes, an area that is both challenging and underexplored. By rigorously dissecting the pipeline into retrieval, join selection, aggregation, and prediction, the study offers clear insights into the interplay between different stages and the resulting performance trade-offs. The development of YADL—a controlled, semi-synthetic data lake—adds a significant experimental asset that allows for systematic exploration of factors such as table redundancy, join quality, and missing data. Moreover, the use of quantitative performance metrics, Pareto frontier analyses, and critical difference plots not only strengthens the experimental evaluation but also provides a transparent view of the resource versus performance trade-offs. The findings that simple retrieval techniques based on exact containment can outperform more elaborate methods in many scenarios, and that tree-based models handle the noise inherent in automated joins better than neural approaches, are both insightful and practically relevant.\n\nWeaknesses: Despite the paper’s thorough experimental analysis, there are several technical limitations that warrant further discussion. First, the study confines itself to single-step join operations, thereby not addressing the complexities associated with multi-hop or chained joins that are common in heterogeneous real-world data lakes; this limitation reduces the generalizability of the findings to more complex relational structures. Second, while Jaccard containment is central to the retrieval methods, its reliance as a universal metric is problematic—especially in cases where columns have low cardinality or when semantic variations exist between join keys—yet the paper provides only a cursory treatment of alternative similarity measures or fuzzy matching techniques that might better capture such nuances. Third, the high computational cost associated with DFS-based aggregation is acknowledged, but the paper does not explore algorithmic optimizations or alternative methods that could alleviate this overhead, leaving a gap in the practical deployment of such approaches. Fourth, the experiments are conducted with a fixed candidate budget (e.g., selecting the top-30 candidates), but there is limited analysis on how varying this parameter would affect both scalability and predictive performance, potentially masking sensitivity issues in candidate selection. Finally, while the authors attempted to compare against several baseline methods, several state-of-the-art alternatives (such as Lazo, ALITE, and others) could not be fully integrated due to implementation challenges, which may constrain the comprehensiveness of the comparative evaluation and limit insights into the broader state of the art.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a comprehensive pipeline for automating feature augmentation in data lakes by decomposing the process into retrieval, join selection, aggregation, and prediction stages. In this work, the authors propose and evaluate multiple methods for each stage: retrieval methods based on Jaccard containment (including Exact Matching, MinHash, Hybrid MinHash, and the more computationally intensive Starmie), various join selectors (such as Highest Containment, Best Single Join, Stepwise Greedy Join, and Full Join), and aggregation techniques ranging from simple “any” and mean-based methods to the more elaborate DFS (Deep Feature Synthesis). The authors conduct an extensive experimental study across several datasets—including several variants of a semi-synthetic data lake (YADL) and Open Data US—to assess prediction performance alongside run-time and memory usage trade-offs. Their results indicate that while simpler, metric-based methods like Exact Matching coupled with straightforward join selection tend to achieve competitive predictive gains with lower computational cost, more complex methods (for instance, Starmie for retrieval or DFS for aggregation) incur significant overhead without commensurate improvements in downstream performance. Additionally, the experiments underscore that tree-based predictors (e.g., CatBoost) are particularly robust in handling the noise and high missingness introduced by automated table joins, compared to their neural network counterparts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7dzp1DiZ3U", "reviewer": "Reviewer_HxpX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The paper provides the source code for reproducibility.\n2. The paper proposes a new formulation of the machine-learning pipeline of multiple tables by expanding beyond feature synthesis and selection. I found that the proposed framework of \"retrieve, merge, and predict\" is interesting and well-suited to the challenges of big data analysis. \n\n### Weaknesses\n\n1. YADL is one of the paper's main contributions, but the paper lacks information regarding the supervised learning tasks. Thus, it is not clear which pipeline steps are \"most\" important for prediction performance (i.e., the ML tasks could be inherently difficult even with perfect selection and joining, or the ML tasks could be trivially simple once good columns have been joined.)\n1. While the setting is about \"limited memory,\" I found that the paper does not make this evidence in the proposed YADL data. For example, the size of the fully joined table might give a reader a rough idea of the challenge and the need to retrieve, merge, and predict the pipeline. In addition, the authors should acknowledge and discuss the work along the lines of big data architecture [1]. This direction of work might be similar to what authors try to achieve.\n1. The conclusion regarding CatBoost is somewhat vague. The paper attributes the performance of CatBoost to the noise (null) and the cardinality of categorical features, but it has not been shown explicitly. In addition, preprocessing is an essential step for most predictive models, but the paper lacks this detail. \n1. The main text of the paper could be more concise. This allowed more detail in the appendix to be in the main text.\n\n### References\n\n[1] M. Rong, D. Gong and X. Gao, \"Feature Selection and Its Use in Big Data: Challenges, Methods, and Trends,\" in IEEE Access, vol. 7, pp. 19709-19725, 2019, doi: 10.1109/ACCESS.2019.2894366.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors investigate how to automate augmenting a user’s base table with features from other tables in a large data lake and then train a predictive model on the augmented table. The paper argues that no work studies the whole pipeline (retrieve, merge, and predict). They introduce a new semi-synthetic data lake called YADL, built by recombining data from the YAGO knowledge base. This controlled dataset allows for a systematic comparison of different augmentation and learning strategies. The authors also explore using an actual data lake, Open Data US, to validate their findings beyond synthetic settings. The experiment measures predictive performance, computational time, and memory usage. Their experiment results show that simple retrieval criteria based on set containment (exact) can work nearly as well as more advanced approaches. In addition, the result shows that CatBoost tolerate the noisy features and missing data introduced by automated table joins better than neural networks or linear baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add more information about YADL, especially the prediction tasks, such as the number of base tables (ML Tasks), training and testing rows, and, if possible, a ceiling performance (if there is a perfect retrieval and merging).\n2. Add more information about preprocessing steps of predictors.\n3. Revise the discussion to be less duplicate with the result section.", "broader_impact_concerns": "No ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZV7MdgdKKN", "reviewer": "Reviewer_DvRn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- **(S1)** Interesting and important practical problem, and the extensive evaluation with the accompanying benchmark is extremely valuable in identifying the important areas of future research.\n- **(S2)** The paper covers appropriate amount of background information to help the reader clearly understand the different aspect of this problem setup, which is useful in understanding the insights presented in this paper.\n\n\n### Weaknesses\n- **(W1)** The present Pareto plots in Figure 2 are somewhat misleading in that it can lead to counterintuitive insights -- for example, in the \"Retrival Method\" column, it appears that \"Exact\" retrieval can some times be faster than the approximate \"Minhash\". This is of course not true, and an artifact that we are averaging over base tables and DLs (I think) and that the downstream classifier training time is included here. So exact search with say Catboost is overall faster than MinHash with ResNet. While this is true, it does not provide a clear ablation of the different components.\n- **(W2)** Viewing Figure 2 (and table 3), it seems like \"Exact\" retrieval and \"Full Join\" selector are Pareto optimal in terms of resources-performances tradeoff. This seems somewhat disappointing. Any discussion in this observation will be useful otherwise the question is \"why bother trying anything else if the exhaustive process is optimal on average\". Is this highlighting an issue with the benchmark or the considered methods for the different steps, or the way the results are aggregated and presented? It seems unlikely that with all the recent methods developed in this area, the conclusion that \"Exact Retrieval + Full Join\" is Pareto-optimal needs better clarification. \n- **(W3)** While discussion on page 10 claims that Minhash and hybrid-Minhash brings substantial improvements, Figure 2 (left) does not necessarily show them as Pareto optimal methods (unless I am missing something); this conclusion does not seem clear from the figures. In that case, there seems to be a bit of a discrepancy between the actual results and the discussion which needs further elaboration in the text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the complete data augmentation pipeline for modeling that involves starting from a base table, and utilizing a data lake to augment the base table with additional columns for each row in the base table, and finally building a machine learning model on that augmented table. While there has been a lot of work analyzing and automating the choices (such as hyperparameters) in the model building stage, this paper focuses on all the choices in the data augmentation stage which involves retrieval of relevant candidates from the data lakes, and the merging of candidates to create the augmented features. Both these steps involve multiple choices. To this end, this paper\n- Presents a benchmark for such pipelines to evaluate the predictive performance/computational cost trade-offs for the different configurations of this pipeline.\n- Evaluates various such pipeline configurations on this benchmarking suite to develop valuable insights in terms of the capabilities of the different configurations, and provide areas that could be most beneficial for future research in this setup.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the presentation of the results can be enhanced to address some of the questions I raised in the Weaknesses above. Maybe the use of relative metrics (before the aggregation across all base tables) can also improve presentation.\n\nBeyond this, I have a couple of questions:\n- **(Q1)** Only considering methods based on Jaccard similarity will of course lead us to a scenario where the join candidates are selected agnostic to the downstream tasks. I think the embedding based matching scheme can incorporate some relevance to downstream tasks. As fuzzy matching is not considered in this work, it is important to discuss this aspect more explicitly to highlight the caveats of the insights formed from the elaborate evaluation.\n- **(Q2)** _[Minor]_ In the initial motivating example, it is not clear what information of the \"users\" if any is being incorporated. Can this be clarified?\n- **(Q3)** It is not clear what kind of \"retrive, merge\" operations would be performed for test examples/rows. Are these operations happening with training and test data combined? The handling of test data should be clearly and explicitly presented in my opinion.\n  - If the test data augmentation is happening along with training data, how is data leakage handled in this setup? \n  - If the test data augmentation happens separately for each test row, how is the consistency of features selected handled for the test rows, and how are the computational costs of retrieval/merge at test time incorporated in the runtime/performance tradeoff results? \n- **(Q4)** _[MINOR]_ It is not clear how the number of failed configurations (across various sizes of the DL and Base Tables) are considered in the averages utilized in Figure 2.", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found that the problem studied is "challenging and underexplored" (Reviewer H1Bb), "interesting and important" (Reviewer DvRn), the proposed framework is "interesting and well-suited to the challenges of big data analysis" (Reviewer HxpX). They also found that the new data lake offers "a significant experimental asset that allows for systematic exploration of factors" (Reviewer H1Bb) and is "extremely valuable in identifying the important areas of future research." (Reviewer DvRn). Reviewer H1Bb also noted that the findings are "insightful and practically relevant".

During the rebuttal, the authors and reviewers had fruitful exchanges around discussion of limitations and future work, clarified additional details about the tasks and settings, discussed nuances of different comparisons, the suboptimality of naive baselines and clarifying where there is room for improvement. I encourage the authors to reflect these discussions in the main paper and include an explicit section on weaknesses of the existing investigation to guide future work.
Decision: Accept (poster)