CardBench: A Benchmark for Learned Cardinality  Estimation in Relational Databases

Authors: Yannis Chronis, Yawen Wang, Yu Gan, Sami Abu-El-Haija, Chelsea Lin, Carsten Binnig, Fatma Ozcan
Primary area: datasets and benchmarks
Keywords: cardinality estimation, zero-shot, fine tuning, databases

Abstract:
Cardinality estimation is crucial for enabling high query performance in relational
databases. Recently learned cardinality estimation models have been proposed
to improve accuracy but there is no systematic benchmark or datasets which
allows researchers to evaluate the progress made by new learned approaches
and even systematically develop new learned approaches. In this paper, we are
releasing a benchmark, containing thousands of queries over 20 distinct real-world
databases for learned cardinality estimation. In contrast to other initial benchmarks,
our benchmark is much more diverse and can be used for training and testing
learned models systematically. Using this benchmark, we explored whether learned
cardinality estimation can be transferred to an unseen dataset in a zero-shot manner.
We trained GNN-based and transformer-based models to study the problem in three
setups: 1-) instance-based, 2-) zero-shot, and 3-) fine-tuned.
Our results show that while we get promising results for zero-shot cardinality estimation on simple single table queries; as soon as we add joins, the accuracy drops.
However, we show that with fine-tuning, we can still utilize pre-trained models
for cardinality estimation, significantly reducing training overheads compared to
instance specific models. We are open sourcing our scripts to collect statistics,
generate queries and training datasets to foster more extensive research, also from
the ML community on the important problem of cardinality estimation and in
particular improve on recent directions such as pre-trained cardinality estimation.

Decision: reject
Reviewer scores: 5, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730714684430

thread_type:
official_review

<thread_object>
creator: reviewer_uy6z

<content>
summary:
The paper presents CardBench, a benchmark for evaluating learned cardinality estimation models in relational databases. CE is vital for optimizing query performance, yet traditional methods often lack accuracy. CardBench offers a diverse set of datasets and queries, featuring hundreds of thousands of queries across 20 real-world datasets, enabling systematic assessments of new CE approaches.
Experiments with GNN and transformer-based models were conducted under three setups: instance-based, zero-shot, and fine-tuning. While zero-shot estimation shows promise for simple queries, its accuracy declines with complex joins. However, fine-tuning pre-trained models can enhance performance significantly, reducing training costs.
The authors emphasize that CardBench will lower barriers for research in learned CE and encourage further exploration from the machine learning community.

strengths:
- CardBench includes a broad range of datasets, 20 distinct real-world datasets, that provide a more comprehensive evaluation framework for learned cardinality estimation models compared to existing benchmarks.
- The benchmark facilitates systematic testing of various learned CE approaches, allowing researchers to assess model performance comprehensively. By open-sourcing the benchmark, query generator, and associated scripts, the authors foster collaboration and further research in the field, lowering barriers for other researchers to experiment with learned cardinality estimation.
- The paper is clearly written and organised.

weaknesses:
- There are several typos that need to be corrected in future revisions.
- It would be beneficial to include more detailed statistical experimental results. While boxplots effectively illustrate the distribution of results, providing exact values enables precise comparisons between data points.
- The descriptions of the GNN and Transformer models used in the experiments are lacking. More detailed explanations would help readers gain better insights into the methodologies.
- The paper should discuss how the performance of other state-of-the-art methods compares on CardBench, which would provide context and strengthen the findings.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730908644

thread_type:
official_review

<thread_object>
creator: reviewer_eaz6

<content>
summary:
This paper proposes the cardbench, a benchmark dataset for cardinality estimation in relational databases, which contains over 20 distinct real-world databases. This proposed dataset shows clear diverse and proposes several baseline models including the GNN and transformer modules to form a benchmark evaluation of this benchmark dataset.

strengths:
+ A large-scale benchmark dataset for Cardinality Estimation
+ Good evaluations and benchmark analyses.
+ Clear organizations and easy to read.

weaknesses:
+ My major concern is about the benchmark contributions to this research field. In the related work section, the authors talked about how existing benchmarks only contain one or two datasets which is insufficient for testing pretraining models. Thus my question is, how does this proposed benchmark test these pretrained models? It seems the authors only test typical GNN or Transformer architectures. 
This dataset seems not to be comprehensive as a benchmark in this research field.
+ Besides, gathering different datasets or re-organizing them also may help the aforementioned problems. Why is this proposed benchmark unique? I am concerned about the realistic usage and if this is a real problem for the community.
+ There are also several minor issues, including the presentations for the Sect. 3 and the table format for table 1. These presentations could be improved.

questions:
Please refer to the weakness section and I am concerned about the realistic usage of this proposed benchmark or it is an ``made" problem.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731120001016

thread_type:
official_review

<thread_object>
creator: reviewer_4nbw

<content>
summary:
The paper introduces CardBench, a benchmark for evaluating learned cardinality estimation models in relational databases. Cardinality estimation is vital for query optimization, but existing models lack a comprehensive benchmark for systematic evaluation. CardBench provides this with extensive queries across 20 real-world databases. The study evaluates GNN and transformer-based models in instance-based, zero-shot, and fine-tuned setups, highlighting challenges in zero-shot accuracy with complex queries but demonstrating the potential of fine-tuning pre-trained models. By releasing scripts and data, the authors encourage further research, showing that pre-trained models can achieve high accuracy with reduced training overhead.

strengths:
S1: Cardinality estimation is critical for the database community.

S2: This paper proposes several datasets, which will be useful for academic and industry communities.

S3: Zero-shot cardinality estimation seems to be useful.

weaknesses:
W1: The details of the datasets are unclear.

W2: The baselines are too weak and lack state-of-the-art and representative cardinality estimation baselines.

W3: The experiments lack detailed analysis of the proposed models regarding the zero-shot setting.

questions:
D1: This paper proposes several synthetic cardinality estimation datasets. But what are the patterns and distributions in these datasets?

D2: Why are these datasets comprehensive and diverse? From a database view, are these proposed datasets complete? It is better to add more justifications.

D3: This paper proposes zero-shot estimation but only investigates limited GNNs. In cardinality estimation, whether sampling-based approaches or summary-based approaches, all work under a zero-shot setting. These approaches do not need any training instances. But this paper does not explore these approaches.

D4: Even for GNN-based baselines, they are too simple. 

D5: How about the training time and inference time compared with the time of the training-free methods? If the training-free methods are better than the proposed method, it is meaningless to propose a pre-trained model.

D6: This paper only explores the performance on cardinality estimation error. But how to prove it can achieve better performance on downstream applications? It is better to deploy this model on a real database system to see its improvements.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731126581996

thread_type:
official_review

<thread_object>
creator: reviewer_vebn

<content>
summary:
This paper proposes a benchmarking framework aimed at cardinality estimation within relational databases, using advanced learning methods like GNNs and Transformers.

strengths:
The paper details a systematic data preparation process, including SQL query generation, dataset statistics calculation, and annotated query graph creation, which offers a replicable approach for dataset-agnostic testing.

The paper’s emphasis on creating a model that can generalize to unseen datasets is a unique and relevant shift in the CE field, considering the increasing need for adaptable models in dynamic data environments.

weaknesses:
1. The authors said that they have collected data from 20 datasets with diverse sources compared with existing benchmarks. But I can't see this comparsion to conclude that how novel this part is. It deserves detailed discussion.

2. Although it includes single-table and multi-table queries, it lacks support for deeply nested and highly complex SQL queries, which are common in real-world database applications. This limitation in query complexity could lead to suboptimal model performance in practical scenarios.

3. Only q-error is used as the main evaluation metric, which may not fully capture model performance. Additional dimensions, such as runtime and resource consumption, could provide a more comprehensive assessment.

questions:
see weakness above
</content>
</thread_object>

</thread>
<thread>
creation date:
1732048721240

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal Response

comment:
We first want to thank the reviewers for the time and effort they put in the reviews. We appreciate the feedback and we will use it to improve our paper. Here, we want to reply to concerns raised in more than one review. We also reply to each of the reviews separately.

We submitted a revised paper to open review to address the points raised in the reviews. All changes are colored in blue.

(A) Before, we discuss the individual comments of the reviewers, we would like to clarify the positioning of the paper:

In this paper the primary contribution is the datasets and queries we executed on those datasets to construct the benchmark. Generating this benchmark is non-trivial and also required significant time, effort, and resources making it a valuable contribution to the research community as labeled training data ( which involves running thousands of queries) is very hard to find for database tasks. With this benchmark, we want to achieve two goals: (1) show that more research is needed to enable zero-shot cardinality estimation and (2) based on the dataset and workloads (with true cardinalities) enable more research in this direction.

To motivate using a zero-shot/pre-trained approach and to show that more research is needed we evaluated two model variants (one based on GNNs and transformers) in a zero-shot manner. For this we used a leave-one-out setting where we used all datasets and workloads except one for training the model and then tested on the left-out dataset and workload the zero-shot capabilities. While the models themselves are variations of standard architectures (e.g. the GNN model uses a new message passing scheme), we use a novel input representation for the datasets and queries to enable zero-shot cardinality prediction on unseen datasets and workloads.  However, the models and how we represent the datasets and workloads are only the secondary focus of this paper, and the models only serve only as the motivation to show the need for a new benchmark on zero-shot cardinality estimation and that this approach requires further research which is enabled by this benchmark.


B) Second, we want to address shared comments regarding the diversity of our datasets and workloads (vebN: W1, 4NBw:D1/D2)


The datasets and workloads (queries) of CardBech are diverse to be suitable to train and test zero-shot and pre-trained models. To quantify diversity we have updated Table 1 with additional data statistics as well as added Table 4 which summarizes the cardinalities of the workload queries. Together Table 1 and 4 illustrate the diversity of the dataset and workloads which constitute CardBench. In more detail:
 
1) In our original submission, Table 1 uses table count, column count, row count and unique value count as diversity metrics for the workloads.  We have updated Table 1 (in a revised submission to open review – changes are colored blue) with the number join paths (how many relationships exist between the tables of each dataset), type of join paths (star, chain, mixed) as well as the average pearson correlation between columns of the same table. 
2) To quantify diversity of our workloads, we added in the appendix Table 4, that lists the max, average and standard deviation of the query cardinalities per dataset for the single table, binary join and multi-join queries. At this moment we added the statistics for the single table, binary join queries and we are calculating the statistics for the multi-join queries which we will add once we have them.

Further, we are planning to list the sources of the datasets in the repository of our benchmark (All datasets we use are publicly available). If you have suggestions for additional diversity statistics we would be happy to extend Table 1 and 4. Note that we are also providing an end-to-end training data preparation pipeline, which can be invoked on more datasets and used to generate different workloads.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732982403425

thread_type:
official_comment

<thread_object>
creator: area_chair_fpfb

<content>
title:
The deadline for Author/Reviewer discussion period is in three days!

comment:
Dear Reviewers,

Thanks again for providing your constructive comments and suggestions. The deadline for the Author/Reviewer discussion period is in three days (December 2). Please make sure to read the authors' responses and follow up with them if you have any additional questions or feedback.

Best,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces CardBench, a benchmark comprising thousands of queries across 20 distinct databases, along with scripts to compute data summary statistics and generate queries. Using this benchmark, the paper evaluates learned cardinality estimation in a zero-shot setting. GNN-based and transformer-based models are trained and analyzed under three setups: (1) instance-based, (2) zero-shot, and (3) fine-tuned.

**Strengths**
- CardBench provides a timely platform that could facilitate future research in cardinality estimation.
- The experimental results reveal some interesting insights into the challenges and performance of current approaches.

**Weaknesses**

- Reviewers are not totally convinced whether the collected datasets are comprehensive enough. 
- The paper only tests typical GNN and transformer based architectures, which are insufficient. Reviewers suggest to show how other state-of-the-art methods compare on CardBench. 

While CardBench has the potential to contribute to the field of cardinality estimation, the major weaknesses, including concerns about dataset comprehensiveness and insufficient evaluation with diverse methods, limit its overall impact. Without addressing these issues, the benchmark’s contribution appears to fall below the bar for acceptance.