DCA-Bench: A Benchmark for Dataset Curation Agents

Authors: Benhao Huang, Yingzhuo Yu, Jin Huang, Xingjian Zhang, Jiaqi W. Ma
Primary area: datasets and benchmarks
Keywords: Dataset Curation, LLM Agent, Automatic Evaluation

Abstract:
The quality of datasets plays an increasingly crucial role in the research and development of modern artificial intelligence (AI). Despite the proliferation of open dataset platforms nowadays, data quality issues, such as incomplete documentation, inaccurate labels, ethical concerns, and outdated information, remain common in widely used datasets. Furthermore, these issues are often subtle and difficult to be detected by rule-based scripts, therefore requiring identification and verification by dataset users or maintainers--a process that is both time-consuming and prone to human mistakes. With the surging ability of large language models (LLM), it’s promising to streamline the discovery of hidden dataset issues with LLM agents. To achieve this, one significant challenge is enabling LLM agents to detect issues in the wild rather than simply fixing known ones. In this work, we establish a benchmark to measure LLM agent’s ability to tackle this challenge. We carefully curate 221 representative test cases from eight popular dataset platforms and propose an automatic evaluation framework using GPT-4. Our proposed framework shows strong empirical alignment with expert evaluations, validated through extensive comparisons with human annotations. Without any hints, a baseline GPT-4 agent can only reveal 11% of the data quality issues in the proposed dataset, highlighting the complexity of this task and indicating that applying LLM agents to real-world dataset curation still requires further in-depth exploration and innovation.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730560645875

thread_type:
official_review

<thread_object>
creator: reviewer_wl5u

<content>
summary:
This paper introduces a novel benchmark for assessing the capability of LM-based curator agents in identifying potential issues within datasets. The authors also streamline the evaluation process, enabling non-expert usage. Baseline evaluations conducted with GPT-4 demonstrate the system’s performance and highlight the potential value of their autonomous dataset curation approach.

strengths:
1. This paper establishes a complete curation agent benchmark derived from real-world datasets, providing test cases ranging from easy to challenging, and potentially can be serve as an important foundation in this area.
2. The paper is well-organized, presenting a clear workflow of the proposed method alongside some demonstrations that effectively validate the robustness of the evaluation framework.

weaknesses:
From my perspective, this paper presents the following limitations:
1. The benchmark appears to rely primarily on data sources from machine learning engineering, with a limited number of test cases. This raises questions about its generalizability to other domains, such as image datasets.
2. While the work presents a well-defined benchmark with a structured testing procedure, it largely resembles a series of experiments conducted with GPT-4 rather than a comprehensive agent-testing framework. The experiments on more agents are insufficiently thorough.
3. The paper does not provide open-source code.

questions:
Can your method be effectively applied to other domains, such as image datasets like ImageNet? Do you intend to make the benchmark publicly available?\
\
\
Additional: \
Pointing out an issue with your paper’s formatting: typically, one page contains around 53 lines of content, but your paper only has 46 lines, like page 1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720209072

thread_type:
official_review

<thread_object>
creator: reviewer_vzxc

<content>
summary:
The paper introduces a Dataset Curation Agents Benchmark to evaluate the ability of LLM agents to identify hidden data quality issues in community-contributed datasets. Focusing on the challenge of detecting undocumented issues rather than solving known problems, this benchmark uses 221 real-world test cases from popular dataset platforms and an automatic evaluation framework utilizing GPT-4, with alignment to expert evaluations. Initial results indicate a baseline GPT-4 Curator detects only a small percentage of issues, underscoring the complexity and need for further research in autonomous dataset curation.

strengths:
1. Innovative Concept: The paper presents a novel approach by shifting focus from issue-solving to issue-discovery in dataset curation.
2. Comprehensive Coverage: DCA-Bench provides a broad testing ground with diverse curated test cases, representing real-world scenarios across different platforms.
3. Automatic Evaluation Framework: The use of GPT-4 for evaluation is a practical advancement, offering scalability where human annotation is not feasible.
4. Clear Practical Relevance: Tackling hidden data quality issues addresses a significant gap in dataset management, which is crucial for improving AI research outcomes.

weaknesses:
1.	Limited Test Set Size: With only 221 samples, and limited instances in some categories (e.g., only 10 ethical instances), the test set might not sufficiently capture the complexity of real-world data quality issues. Increasing the dataset size and diversity could improve robustness.
2.	Missing Related Works: The paper does not adequately engage with existing literature regarding data system and LLM agents, such as [arXiv.2402.02643, LLM-Enhanced Data Management], [SIGMOD’24, Data-juicer: A one-stop data processing system for large language models], and [ICML'24, DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning], which could provide valuable context and highlight the novelty of this work.
3.	Clarity Issues and minor writing suggestions: There are several presentation issues, such as missing references to Figure 1 in the text and unclear definitions in Table 1 for sample-level, type-level, and tag-level insights. Addressing these could enhance readability and comprehension. Beside, Line 104-105 lacks a “:” Line 213~214 lacks the references of “Kaggle、OpenML、TF-Dataset、Open-Data-Registry, Five-Thirty-Eight”.
4.	Applicability to different types of datasets: only eight open data set platforms have been mentioned, but there is a lack of analysis on whether different types of data sets (such as medical data, financial data, etc., have special properties) are equally applicable. It is recommended to add discussion or preliminary experimental results to demonstrate the versatility of DCA-Bench.

questions:
None, plz see the weakness above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730786646941

thread_type:
official_review

<thread_object>
creator: reviewer_jovr

<content>
summary:
The work introduces a benchmark aimed at evaluating the ability of large language model (LLM) agents to autonomously detect subtle data quality issues in large, real-world datasets.

The work addresses the ongoing challenges in maintaining data quality on open platforms, where issues like mislabeling, documentation gaps, and ethical concerns are prevalent.

DCA-Bench comprises 221 real-world test cases across eight major dataset platforms and employs an automatic evaluation framework using GPT-4 to assess the performance of LLMs as dataset curators.

strengths:
- The work addresses a significant problem in AI, as dataset quality is critical for robust model performance and reliable research outcomes.
- The work provides a foundation for future research into fully autonomous dataset curation systems, which could save time and reduce errors in data management.
- The framework offers different hint levels to evaluate LLMs at multiple stages of problem discovery, providing nuanced insights into model capabilities.
- DCA-Bench constructs an automatic evaluation pipeline: leverages GPT-4 to automate evaluations, which aligns well with expert human assessments, making the process scalable and consistent.

weaknesses:
- There is a lack of a fully realistic testing environment. While comprehensive, the test cases may not fully represent the diversity of data quality issues encountered in real-world curation.
- Performance is heavily dependent on hints. Without hints, the baseline LLM agent detected only 11% of issues, indicating a limited ability to autonomously identify dataset problems.
- The automatic evaluation pipeline, while effective, may introduce subtle biases that differ from nuanced human judgment.

questions:
The benchmark is primarily text-focused, will the benchmark be scalable to multimodal data (e.g., image or audio datasets)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731290328786

thread_type:
official_review

<thread_object>
creator: reviewer_pfuz

<content>
summary:
This paper aims to solve the detection of data quality issues such as data errors, documentation issues, file discrepancies, and legal/ethical risks. This study of this paper starts from curating 221 test cases from eight popular dataset publishing platforms (like HuggingFace, Kaggle) and propose an automatic evaluation framework using GPT-4. It is interesting to know that without any hints, a baseline GPT-4 Curator agent can only reveal 11% of the data quality issues in the 221 test cases. When given the most specific hint, 70% of the issues can be detected.  This is understandable because finding the hidden issues of data quality is a complex task, even for human experts like dataset users and platform maintainers.

strengths:
S1: the exploration of data quality issues is an interesting and important problem. If LLMs are able to automate the detection of data quality issues, they will be helpful for automatic maintenance of data publishing platform.
S2:  The dataset collection process seems reasonable. Four different types of issues are annotated, representing the typical issues that may happen in public data platforms.
S3: The evaluation system leverages LLMs in multiple perspectives for implementing different evaluation strategies, using RAG, code interpreter to write and execute programs, etc.

weaknesses:
W1: the Evaluator is designed to leverage LLMs as judges for assessing whether the detection results are correct or not. However, as authors mentioned:  the annotated test data for the Evaluator are collected after the prompt design for the Evaluator. This raises concerns that the test data may simply align with the designed prompts, potentially indicating that the Evaluator’s performance is optimized for this specific data. This brings into question  about how  the proposed prompt design of the Evaluator can generalize across different Curators and issues in the DCA-Bench.

W2: The performance of the system in identifying dataset quality issues appears to have been evaluated solely on datasets that are known to have issues. It is currently unclear how the system performs on high-quality datasets where no issues are present. This raises concerns about its ability to avoid false positives in scenarios where no data issues exist.

questions:
Are high-quality datasets included in the evaluation process? How about false positives (i.e., incorrect identification of issues in clean datasets) if using the proposed detection models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732559877300

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

As the conclusion of our rebuttal period approaches, I wanted to kindly check if our rebuttal has addressed your concerns. We greatly value your thoughtful feedback and have incorporated clarifications and updates into our drafts accordingly.

If you have any additional concerns, please do not hesitate to let us know at your earliest convenience, so we can strive to respond before the discussion period concludes.

Thank you very much for your time and consideration.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces the DCA-Bench to evaluate the ability of large language model (LLM) agents to detect data quality issues. DCA-Bench constructs an automatic evaluation pipeline by leveraging GPT-4 for automation. This research focuses on a vital issue, and some experimental results provide valuable insights. However, the primary challenge lies in the ubiquitous nature of detecting data quality issues for LLMs. Given the scale and complexity of this problem, the small-sized DCA-Bench offers only limited contributions. To enhance its impact, further exploration of domain-specific data is necessary.