Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond

## Abstract

Abstract Large-scale data collection is essential for developing personalized training data, mitigating the shortage of training data, and fine-tuning specialized models. However, creating high-quality datasets quickly and accurately remains a challenge due to annotation errors, the substantial time and costs associated with human labor. To address these issues, we propose Automatic Dataset Construction (ADC), an innovative methodology that automates dataset creation with negligible cost and high efficiency. Taking the image classification task as a starting point, ADC leverages LLMs for the detailed class design and code generation to collect relevant samples via search engines, significantly reducing the need for manual annotation and speeding up the data generation process. Despite these advantages, ADC also encounters real-world challenges such as label errors (label noise) and imbalanced data distributions (label bias). We provide open-source software that incorporates existing methods for label error detection, robust learning under noisy and biased data, ensuring a higher-quality training data and more robust model training procedure. Furthermore, we design three benchmark datasets focused on label noise detection, label noise learning, and class-imbalanced learning. These datasets are vital because there are few existing datasets specifically for label noise detection, despite its importance. Finally, we evaluate the performance of existing popular methods on these datasets, thereby facilitating further research in the field.

## 1 Introduction

In the era of Large Language Models (LLMs), the literature has observed an escalating demand for fine-tuning specialized models [, , ], highlighting the urgent need for customized datasets [, , ].

Traditional Dataset Construction (TDC) typically involves sample collection followed by labor-intensive annotation, requiring significant human efforts [, , , ]. Consequently, TDC is often hindered by the limitations of human expertise, leading to suboptimal design [], data inaccuracies [, , , , ], and extensive manual labor [, ]. Furthermore, certain datasets are inherently challenging or risky to collect manually, such as those for fall detection in elderly individuals, dangerous activities like extreme sports, and network intrusion detection. Therefore, there is a growing need for more automated and efficient data collection methods to enhance accuracy and efficiency in dataset creation [, , ]. To address these challenges, we propose the Automatic Dataset Construction (ADC), an innovative approach designed to construct customized large-scale datasets with minimal human involvement. Our methodology reverses the traditional process by starting with detailed annotations that guide sample collection. This significantly reduces the workload, time, and cost associated with human annotation, making the process more efficient and targeted for LLM applications, ultimately outperforming traditional methods.

Figure: Figure 1: Comparisons of key steps in dataset construction. In Step 1: Dataset design, ADC utilizes LLMs to search the field and provide instant feedback, unlike traditional methods that rely on manual creation of class names and refine through crowdsourced worker feedback. In Step 2: Labeling, ADC reduces human workload by flipping the data collection process, using targets to search for samples. In Step 3: Cleaning, ADC instructs human labor to filter noisy labeled samples from previous steps, instead of relabeling.
Refer to caption: extracted/5796241/fig/ADC_Fig_1.png

##### Traditional-Dataset-Construction v.s. Automatic Dataset Construction

Figure 1 illustrates the difference between Traditional Dataset Construction (TDC) and Automatic Dataset Construction (ADC). TDC typically unfolds in two stages: developing classification categories and employing human labor for annotation. Creating comprehensive categories requires deep domain knowledge and experience, tasks that even expert researchers find challenging []. Crowdsourcing is often used to refine these categories, but it increases time and costs without necessarily improving label quality [, ]. Annotation by human workers introduces label noise, which impacts dataset reliability, even when multiple inputs are aggregated []. In contrast, ADC offers improvements at each key step. In the “Dataset design", ADC uses LLMs to automate field searches and provide instant feedback, unlike traditional manual class and attribute creation. In the sample annotation steps, ADC reverses the labeling process by using predefined targets to search for samples, human annotators are then instructed to filter noisy labeled samples, significantly reducing the need for costly human annotation.

Our main contributions can be summarized as follows:

- $\bullet$
The Automatic-Dataset-Construction (ADC) pipeline: We introduce Automatic-Dataset-Construction (ADC), an automatic data collection approach that requires minimal human efforts, tailored for the specialized large-scale data collection. Leveraging ADC, we developed Clothing-ADC, an image dataset containing one million images with over 1,000 subclasses for each clothing type. Our dataset offers a rich hierarchy of categories, creating well-defined sub-populations that support research on a variety of complex and novel tasks.
- $\bullet$
Software efforts for addressing dataset construction challenges:
We explore several challenges observed in real-world dataset construction, including detecting label errors, learning with noisy labels, and class-imbalanced learning. To improve the quality of the constructed data and model training, we provide well-written software that incorporates existing solutions to these challenges.
- $\bullet$
Benchmark efforts and Comprehensive Service: To further facilitate the exploration of the aforementioned challenges (label noise detection and learning, class-imbalanced learning), we customize three benchmark subsets and provide benchmark performances of the implemented methods in our software. This offers researchers a platform for performance comparisons, enhancing the evaluation and refinement of their approaches.

## 2 Automatic-Dataset-Construction (ADC)

In this section, we discuss the detailed procedure of ADC, as well as an empirical application.

### 2.1 The ADC pipeline

The ADC pipeline generates datasets with finely-grained class and attribute labels, utilizing data diagnostic software to perform data curation. Below, we provide a step-by-step guide to collecting the Clothing-ADC, a clothes image dataset, along with an overview of its statistics and key information. The overall Automatic-Dataset-Construction (ADC) pipeline is illustrated in Figure 1.

##### Step 1: Dataset design with large language models (LLM)

- $\bullet$
Customization: The LLM is tailored to understand specific domains (i.e., fashion) via In-Context-Learning based prompts to leverage its pre-existing knowledge, or fine-tuning on the customized dataset. This enables the LLM to significantly simplify the design phase, making the task easier for designers by allowing them to select from existing options rather than creating from scratch. Given the extensive availability of "clothes"-related domain knowledge on the internet, we skip this procedure for Clothing-ADC.
- $\bullet$
Detailed categories identification: LLMs assist researchers in conducting a more thorough search in the field by processing and analyzing numerous concepts simultaneously, unlike humans who may overlook certain factors when faced with a large volume of concepts []. We utilize LLMs to identify attribute types for each class. Then use a prompt of "Show me <30-80> ways to describe <Attribute> of <Class>" to generate the proposed subclasses.
- $\bullet$
Iterative refinement: The initial category list generated by the LLM undergoes review and refinement either by domain experts or through self-examination by the LLM itself, ensuring alignment with specific application or research needs, as shown in Figure 2. This iterative refinement process enables the creation of a high-quality dataset with finely-grained class labels. Additionally, this approach facilitates rapid iterative feedback during the design phase, offering a significant advantage over traditional methods that rely on annotator feedback during the test run annotation phase. This acceleration enables researchers to explore and refine their ideas more efficiently, resulting in better dataset quality and reduced development time.

Figure: Figure 2: Example of using LLM for iterative refinement of attribute designs.
Refer to caption: extracted/5796241/fig/Iterative_design_2.png

##### Step 2: Automated querying and downloading

For image data collection, ADC utilizes APIs provided by Google Images or Bing Images for automated querying. Each category and attribute identified in the first step can be used to formulate search queries.

##### Step 3: Data curation and cleaning

Implement or use existing data curation software capable of identifying and filtering out irrelevant images, such as Docta [], CleanLab [], and Snorkel [], etc. For example, these tools can identify when an item is mislabeled regarding its type, material, or color. Finally, ADC aggregates the suggested labels recommended by the dataset curation software and removes potentially mislabeled or uncertain samples. For illustration, we adopt a data-centric label curation software (Docta) in Algorithm 1.

The high-level idea of this algorithm is to estimate the essential label noise transition matrix $T_Est$ without using ground truth labels, achieved through the consensus equations (Part A). Following this, Algorithm 1 identifies those corrupted instances via the cosine similarity ranking score among features as well as a well-tailored threshold based on the obtained information (i.e., $T_Est$), and then relabels these instances using KNN-based methods (Part B). For more details, please refer to work [, , ].

Figure: Algorithm 1 Data centric curation (Docta)

### 2.2 Clothing-ADC

To illustrate the ADC pipeline, we present the Clothing-ADC dataset, which comprises a substantial collection of clothing images. The dataset includes 1,076,738 samples, with 20,000 allocated for evaluation, another 20,000 for testing, and the remaining samples used for training. Each image is provided at a resolution of 256x256 pixels. The dataset is categorized into 12 primary classes, encompassing a total of 12,000 subclasses, with an average of 89.73 samples per subclass. Detailed statistics of the dataset are provided in Table 1. The following subsection elaborates on the dataset construction process in comprehensive detail.

Subclass design Utilizing GPT-4, we identified numerous attribute options for each clothing type. For example, in the case of sweaters, we recognized eight distinct attributes: color, material, pattern, texture, length, neckline, sleeve length, and fit type. The language model was able to find 30-50 options under each attribute. Our Clothing-ADC dataset includes the three most common attributes: color, material, and pattern, with each attribute having ten selected options. This results in 1000 unique subclasses per clothing type. The selected attributes are detailed in Table 6 (Appendix).

Figure: Table 1: Dataset information summary of Clothing-ADC Dataset.
Refer to caption: extracted/5796241/fig/dataset_sample.png

Data collection The ADC pipeline utilizes the Google Image API to collect clothing images by formulating queries that include attributes such as "Color + Material + Pattern + Cloth Type" (e.g., "white cotton fisherman sweater"). Figure 3 shows examples of these queries and the corresponding images retrieved. The relevance of the search results tends to decline after a significant number of samples are gathered, leading us to set a cutoff threshold of 100 samples per query. After removing broken links and improperly formatted images, each subclass retained approximately 90 samples. These queries generated noisy, webly-labeled data for the training set.

Creating a test set Note that the collected samples may suffer from web-based label noise, where annotations might be incorrect due to mismatches provided by search engines, the traditional approach typically involves manually re-labeling existing annotations and aggregating multiple human votes per label to ensure a high-quality subset for testing purposes. Our ADC pipeline enhances efficiency by presenting annotators with a set of samples that share the same machine-generated label. Annotators are then tasked with selecting a subset of correctly labeled samples, choosing a minimum of four samples out of twenty. This method significantly reduces both manual effort and difficulty, encouraging annotators to critically evaluate machine-generated labels and thereby reducing the effect of human over-trust in AI answers [, ]. The samples selected through this process are considered “clean" labels, representing a consensus between human judgment and machine-generated labels [].

Table 2 provides an insightful comparison between existing datasets and Clothing-ADC. Briefly speaking, compared with existing datasets, the ADC pipeline is able to help humans without domain expertise to create fine-grained attributes for the dataset, and automatic annotation and label cleaning drastically eliminate human effort during label creation.

**Table 2: Compare with existing datasets Our ADC pipeline creates a large-scale image classification dataset with a clean test set. Most existing datasets require human effort for labeling, whereas our pipeline can automatically annotate and clean the data. While Clothing-ADC provides fine-grained attribute labels, our dataset design does not require human expertise in the field.**
| Dataset | # Train/Test | # Classes | Noise Rate(%) | Has Attributes | Auto annotation | Require expert? |
| --- | --- | --- | --- | --- | --- | --- |
| iNaturalist [] | 579k/279k | 54k | Close to 0 | ✗ | ✗ | ✓ |
| WebVision [] | 2.4M/100k | 1000 | 20 | ✗ | ✓ | ✓ |
| ANIMAL-10N [] | 50k/10k | 10 | 8 | ✗ | ✗ | ✗ |
| CIFAR-10N [] | 50k/10k | 10 | 9.03/25.60/40.21 | ✗ | ✗ | ✗ |
| CIFAR-100N [] | 50k/10k | 100 | 25.6/40.2 | ✗ | ✗ | ✗ |
| Food-101N [] | 75.75k/25.25k | 101 | 18.4 | ✗ | ✗ | ✓ |
| Clothing1M [] | 1M in all | 14 | 38.5 | ✗ | ✗ | ✓ |
| Clothing-ADC (Ours) | 1M/20k | 12 | 22.2-32.7 | 12k | ✓ | ✗ |

## 3 Challenge one: dealing with imperfect data annotations

The first pervasive and critical challenge during the automatic dataset construction lies in the prevalence of noisy/imperfect labels. This issue is intrinsic to web-sourced data, which, although rich in diversity, often suffers from inaccuracies due to the uncurated nature of the internet. These errors manifest as mislabeled images, inconsistent tagging, and misclassified attributes, introducing non-negligible noise into the dataset that may adversely affect the training and performance of machine learning models. The following discussion bridges the gap between imperfect data and curated data via mining and learning with label noise, to refine data quality, enhance label accuracy, and ensure the reliability of Auto-Dataset-Construction (ADC) for high-stakes AI applications.

##### Formulation

Let $D:=\{(x_{n},y_{n})\}_{n\in[N]}$ represent the training samples for a $K$-class classification task, where $[N]:=\{1,2,...,N\}$. Suppose that these samples $\{(x_{n},y_{n})\}_{n\in[N]}$ are outcomes of the random variables $(X,Y)\in\mathcal{X}\times\mathcal{Y}$, drawn from the joint distribution $\mathcal{D}$. Here, $\mathcal{X}$ and $\mathcal{Y}$ denote the spaces of features and labels, respectively. However, classifiers typically access a noisily labeled training set $\widetilde{D}:=\{(x_{n},\tilde{y}_{n})\}_{n\in[N]}$, assumed to arise from random variables $(X,\widetilde{Y})\in\mathcal{X}\times\widetilde{\mathcal{Y}}$, drawn from the distribution $\widetilde{\mathcal{D}}$. It is common to observe instances where $y_{n}\neq\tilde{y}_{n}$ for some $n\in[N]$. The transition from clean to noisy labels is typically characterized by a noise transition matrix $T(X)$, defined as $T_{i,j}(X):=\mathbb{P}(\widetilde{Y}=j\mid Y=i,X)$ for all $i,j\in[K]$ [, , ].

### 3.1 The challenge of label noise detection

While employing human annotators to clean data is effective in improving label quality, it is often prohibitively expensive and time-consuming for large datasets. A practical alternative is to enhance label accuracy automatically by first deploying algorithms to detect potential errors within the dataset and then correcting these errors through additional algorithmic processing or crowdsourcing.

#### 3.1.1 Existing approaches to detect label noise

##### Learning-Centric Approaches:

Learning-centric approaches often leverage the behavior of models during training to infer the presence of label errors based on how data is learned. One effective strategy is confidence-based screening, where labels of training instances are scrutinized if the model’s prediction confidence falls below a certain threshold. This approach assumes that instances with low confidence scores in the late training stage are likely mislabeled []. Another innovative technique involves analyzing the gradients of the training loss w.r.t. input data. Pruthi et al. [] utilize gradient information to detect anomalies in label assignments, particularly focusing on instances where the gradient direction deviates significantly from the majority of instances. Researchers have also utilized the memorization effect of deep neural networks, where models tend to learn clean data first and only memorize noisy labels in the later stages of training. Techniques that track how quickly instances are learned during training can thus identify noisy labels by focusing on those learned last [, , ].

##### Data-Centric Approaches:

Data-centric methods focus on analyzing data features and relationships rather than model behavior for detection. The ranking-based detection method [] ranks instances by the likelihood of label errors based on their alignment with model predictions. An ensemble of classifiers evaluates each instance, flagging those that consistently deviate from the majority vote as noisy. Neighborhood Cleaning Rule [] uses the $k$-nearest neighbors algorithm to check label consistency with neighbors, identifying instances whose labels conflict with the majority of their neighbors as potentially noisy. Zhu et al. [] propose advanced data-centric strategies for detecting label noise without training models. Their local voting method uses neighbor consensus to validate label accuracy, effectively identifying errors based on agreement within the local feature space.

#### 3.1.2 Clothing-ADC in label noise detection

We prepared a subset of 20,000 samples from the Clothing-ADC dataset for the label noise detection task, including both noisy and clean labels. We collected three human annotations for each image via Amazon MTurk. Annotators were instructed to classify the labels as correct, unsure, or incorrect. Each sample received three votes. Based on these annotations, we determined the noise rate to be 22.2%-32.7%. Using majority vote aggregation implies uncertainty of the label correctness. By using a more stringent aggregation criterion, more samples are considered as noisy labeled. Under the extreme case where any doubts from any human annotator can disqualify a sample, our auto collected dataset still retains 61.3% of its samples. For a detailed distribution of human votes, see Table 7 in the Appendix.

##### Benchmark efforts

**Table 3: $F_{1}$-Score comparisons among several label noise detection methods on Clothing-ADC.**
| Methods | CORES [] | CL [] | Deep $k$-NN [] | Simi-Feat [] |
| --- | --- | --- | --- | --- |
| $F_{1}$-Score | 0.4793 | 0.4352 | 0.3991 | 0.5721 |

Detection performance comparisons of certain existing solutions are given in Table 3. We adopt ResNet-50 [] as the backbone model to extract the feature here. For each method, we use the default hyper-parameter reported in the original papers. All methods are tested on 20,000 points and predict whether the data point is corrupted or not. We follow [] to apply the baseline methods to our scenario. In Table 3, the performance is measured by the $F_{1}$-score of the detected corrupted instances, which is the harmonic mean of the precision and recall, i.e.,
$F_{1}=\frac{2}{\text{Precision}^{-1}+\text{Recall}^{-1}}.$
Let $v_{n}=1$ indicate that the $n$-th label is detected as a noisy/wrong label, and $v_{n}=0$ otherwise. Then, the precision and recall of detecting noisy labels can be calculated as:
$\text{Precision}=\frac{\sum_{n}\mathbbm{1}(v_{n}=1,\tilde{y}_{n}\neq y_{n})}{
\sum_{n}\mathbbm{1}(v_{n}=1)},\quad\text{Recall}=\frac{\sum_{n}\mathbbm{1}(v_{
n}=1,\tilde{y}_{n}\neq y_{n})}{\sum_{n}\mathbbm{1}(\tilde{y}_{n}\neq y_{n})}.$

### 3.2 The challenge of learning with noisy labels

Another technique is robust learning that can effectively learn from noisy datasets without being misled by incorrect labels, thus maintaining high accuracy and reliability in real-world applications.

#### 3.2.1 Existing approaches to learn with label noise

In this subsection, we contribute to the literature with robust learning software; all covered methods can be mainly summarized into the following three categories: robust loss functions, robust regularization techniques, and multi-network strategies.

##### Robust loss designs

Loss Correction modifies the traditional loss function to address label noise by incorporating an estimated noise transition matrix, thereby recalibrating the model’s training focus []. Loss-Weighting strategies mitigate the impact of noisy labels by assigning lower weights to likely mislabeled instances, reducing their influence on the learning process [, ]. Symmetric Cross-Entropy Loss balances the contributions of correctly labeled and mislabeled instances, improving the model’s resilience to label discrepancies []. Generalized Cross-Entropy Loss, derived from mean absolute error, offers enhanced robustness against outliers and label noise []. Peer Loss Functions form a family of robust loss functions [, , ], leveraging predictions from peer samples as regularization to adjust the loss computation, thereby increasing resistance to noise.

##### Robust Regularization Techniques

Regularization techniques are designed to constrain or modify the learning process, thereby reducing the model’s sensitivity to label noise. Mixup [] generates synthetic training examples by linearly interpolating between pairs of samples and their labels, enhancing model generalization and smoothing label predictions. Label Smoothing [, ] combats overconfidence in unreliable labels by adjusting them towards a uniform distribution. Negative Label Smoothing [] refines this approach by specifically adjusting the smoothing process for negative labels, preserving model confidence in high-noise environments. Early-Learning Regularization tackles the issue of early memorization of noisy labels by dynamically adjusting regularization techniques during the initial training phase [, ].

##### Multi-network strategies

Employing multiple networks can enhance error detection and correction through mutual agreement and ensemble techniques. In Co-teaching, two networks concurrently train and selectively share clean data points with each other, mitigating the memorization of noisy labels []. MentorNet [] equips a student network with a curriculum that emphasizes samples likely to be clean, as determined by the observed dynamics of a mentor network. DivideMix leverages two networks to segregate the data into clean and noisy subsets using a mixture model, allowing for targeted training on each set to manage label noise better [].

#### 3.2.2 Clothing-ADC in label noise learning

**Table 4: Experiment results of label noise learning methods on Clothing-ADC and Clothing-ADC (tiny). We report the model prediction accuracy on the held-out clean labeled test set for comparisons.**
| Methods / Dataset | Clothing-ADC | Clothing-ADC (tiny) |
| --- | --- | --- |
| Cross-Entropy | 74.76 | $67.72\pm 0.40$ |
| Backward Correction [] | 77.51 | $70.49\pm 0.06$ |
| Forward Correction [] | 78.45 | $70.60\pm 0.14$ |
| (Positive) LS [] | 81.94 | $70.67\pm 0.15$ |
| (Negative) LS [] | 78.65 | $70.14\pm 0.13$ |
| Peer Loss [] | 78.58 | $70.92\pm 0.17$ |
| $f$-Div [] | 77.43 | $68.98\pm 0.22$ |
| Divide-Mix [] | 77.00 | $71.58\pm 0.11$ |
| Jocor [] | 78.47 | $72.81\pm 0.02$ |
| Co-Teaching [] | 80.49 | $70.55\pm 0.08$ |
| LogitCLIP [] | 77.85 | $70.16\pm 0.14$ |
| TaylorCE [] | 81.87 | $71.11\pm 0.07$ |

We provide two versions of the Label Noise Learning task, Clothing-ADC and Clothing-ADC (tiny). Specifically, Clothing-ADC leverages the whole available (noisy) training samples to construct the label noise learning task. The objective is to perform class prediction w.r.t. 12 clothes types:
Sweater, Windbreaker, T-shirt, Shirt, Knitwear, Hoodie, Jacket, Suit, Shawl, Dress, Vest, Underwear. We also provide a tiny version of Clothing-ADC, which contains 50K training images, sharing similar size with certain widely-used ones, i.e., MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, etc.

##### Estimated noise level of Clothing-ADC

We selected a subset of 20,000 training samples and asked human annotators to evaluate the correctness of the auto-annotated dataset. After aggregating three votes from annotators, we estimate the noise rate to be 22.2%-32.7%, which consists of 10.5% of the samples having ambiguity and 22.2% being wrongly labeled. The remaining 77.8% of the samples were correctly labeled. The detailed distribution of human votes is given in Appendix Table 7.

##### Benchmark efforts

In this task, we aim to provide the performance comparison among various learning-with-noisy-label solutions. All methods utilize ResNet-50 as the backbone model and are trained for 20 epochs to ensure a fair comparison. We report the model prediction accuracy on the held-out clean labeled test set. For the tiny version, we conduct three individual experiments using three different random seeds and calculate the mean and standard deviation. As shown in Table 4, certain methods, such as Positive LS and Taylor CE, significantly outperform Cross-Entropy. These results underscore the importance and necessity of pairing ADC with robust learning software.

## 4 Challenge two: dealing with imbalanced data distribution

We now discuss another real-world challenge: when imperfect annotations meet with imbalanced class/attribute distributions. As shown in Figure 4, long-tailed data distribution is a prevalent issue in web-based datasets: to collect a dataset of wool suits without a specified target color on Google Image, the majority would likely be dark or muted shades (grey, black, navy), with few samples in brighter colors like pink or purple. This natural disparity results in most data points belonging to a few dominant categories, while the remaining are spread across several minority groups.

Figure: Figure 4: Long-tailed data distribution is a prevalent issue in many datasets. Searching “wool suit" in Google image results in dark wool suits, while only a few are of a light color (red/pink).
Refer to caption: extracted/5796241/fig/longtail_insight_2.png

We are interested in how class-imbalance intervenes with learning. In real-world scenarios, the distribution of classes tends to form a long-tail form, in other words, the head class and the tail class differ significantly in their sample sizes, i.e., $\max_{k}\mathbb{P}(Y=k)\gg\min_{k^{\prime}}\mathbb{P}(Y=k^{\prime})$.

### 4.1 Existing approaches for class imbalance learning

##### Data-Level Methods

Data-level methods modify training data to balance class distribution, focusing on adjusting the dataset by increasing minority class instances or decreasing majority class instances. Oversampling increases the number of minority class instances to match or approach the majority class. This can be done through simple duplication [] (e.g., random oversampling) or generating synthetic data [, , , ]. Undersampling reduces the number of majority class instances, helping to balance class distributions but potentially discarding useful information [, , ].

##### Algorithm-Level Methods

These methods adjust the training process or model to handle unequal class distributions better. Specifically, cost-sensitive learning assigns different costs to misclassifications of different classes, imposing higher penalties for errors on the minority class []. It modifies the loss function to incorporate misclassification costs, encouraging the model to focus more on minority class errors [, ]. Thresholding adjusts the decision threshold for class probabilities to account for class imbalance. Instead of using a default threshold, different thresholds are applied based on class distribution, modifying the decision process for predicting class labels [, ].

### 4.2 Clothing-ADC in class-imbalanced learning

Note that in the label noise learning task, the class distributes with almost balanced prior. However, in practice, the prior distribution is often long-tail distributed. Hence, the combined influence of label noise and long-tail distribution is a new and overlooked challenge presented in the literature. To facilitate the exploration of class-imbalanced learning, we tried to reduce the impact of noisy labels via selecting high-quality annotated samples as recognized by dataset curation software. Human estimation suggested a noise rate of up to 22.2%, and 10.5% marked as uncertain. To address this, we employed two methods to remove noisy samples: a data centric curation (Algorithm 1), which removed 26.36% of the samples, and a learning-centric curation (Appendix Algorithm 3), which removed 25%. Combined, these methods eliminated 45.15% of the samples, with an overlap of 6.21% between the two approaches. We provide Clothing-ADC CLT, which could be viewed as the long-tail (class-level) distributed version of Clothing-ADC. Denote by $\rho$ the imbalanced ratio between the maximum number of samples per class and the minimum number of samples per class. In practice, we provide $\rho=10,50,100$ (class-level) long-tail version of Clothing-ADC.

##### Benchmark efforts

Regarding the evaluation metric, we follow from the recently proposed metric [], which considers an objective that is based on the weighted sum of class-level performances on the test data, i.e., $\sum_{i\in[K]}g_{i}\text{Acc}_{i}$, where $\text{Acc}_{i}$ indicates the accuracy of the class $i$:

$$ $\displaystyle\delta\textbf{-worst accuracy:}\qquad\min_{g\in\Delta_{K}}~{}~{} \sum\nolimits_{i\in[K]}g_{i}\text{Acc}_{i},~{}~{}~{}\text{s.t. }D(\mathbf{g}, \mathbf{u})\leq\delta\,.$ $$

Here, $\Delta_{K}$ denotes the $(K-1)$-dimensional probability simplex, where $K$ is the number of classes as previously defined. Let $\mathbf{u}\in\Delta_{K}$ be the uniform distribution, and $\mathbf{g}:=[g_{1},g_{2},...,g_{K}]$ is the class weights. The $\delta$-worst accuracy measures the worst-case $\mathbf{g}$-weighted performance with the weights constrained to lie within the $\delta$-radius ball around the target (uniform) distribution. For any chosen divergence $D$, it reduces to the mean accuracy when $\delta=0$ and to the worst accuracy for $\delta\to\infty$. The objective interpolates between these two extremes for other values of $\delta$ and captures our goal of optimizing for variations around target priors instead of more conventional objectives of optimizing for either the average accuracy at the target prior or the worst-case accuracy.

Different from the previous dataset we used in noise learning, we use a cleaner dataset for this class-imbalance learning to avoid the distractions of noisy labels. The size of this dataset consists of 56,2263 images rather than 1M. The backbone model we use is ResNet-50. For the class distributions for different $\rho$, we include them in the Appendix. All the experiments are run for 5 times and we calculate the mean and standard deviation. With the imbalance ratio going larger, the accuracy becomes worse, which is expected for a more difficult task.

**Table 5: $\delta$-worst accuracy of class-imbalanced learning baselines on clothing-ADC CLT dataset.**
| Method | $\delta=0$ Worst Accuracy | $\delta=1$ Worst Accuracy | $\delta=\infty$ Worst Accuracy |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $\rho=10$ | $\rho=50$ | $\rho=100$ | $\rho=10$ | $\rho=50$ | $\rho=100$ | $\rho=10$ | $\rho=50$ | $\rho=100$ |
| Cross Entropy | $57.80\pm 0.25$ | $33.85\pm 0.13$ | $30.10\pm 0.22$ | $19.79\pm 0.23$ | $0.35\pm 0.11$ | $0.00\pm 0.00$ | $0.96\pm 0.26$ | $0.00\pm 0.00$ | $0.00\pm 0.00$ |
| Focal [] | $72.70\pm 0.19$ | $65.17\pm 0.29$ | $62.28\pm 0.31$ | $49.66\pm 1.09$ | $34.14\pm 1.05$ | $29.12\pm 0.92$ | $38.12\pm 1.76$ | $19.46\pm 1.49$ | $13.44\pm 1.73$ |
| LDAM [] | $72.50\pm 0.15$ | $65.70\pm 0.26$ | $63.25\pm 0.35$ | $51.13\pm 0.78$ | $36.86\pm 1.03$ | $30.88\pm 1.07$ | $40.90\pm 1.53$ | $23.24\pm 1.69$ | $15.69\pm 2.13$ |
| Bal-Softmax [] | $74.18\pm 0.08$ | $70.48\pm 0.55$ | $69.47\pm 0.44$ | $56.57\pm 0.93$ | $53.37\pm 2.31$ | $44.24\pm 2.83$ | $48.54\pm 2.27$ | $45.64\pm 3.98$ | $50.60\pm 1.40$ |
| Logit-Adjust [] | $74.08\pm 0.05$ | $70.94\pm 0.24$ | $69.44\pm 0.18$ | $56.00\pm 1.39$ | $53.93\pm 2.46$ | $49.70\pm 2.64$ | $47.45\pm 2.26$ | $47.76\pm 4.07$ | $43.26\pm 4.69$ |
| Post-hoc [] | $62.54\pm 0.11$ | $54.84\pm 0.15$ | $49.63\pm 0.71$ | $35.67\pm 0.49$ | $24.14\pm 1.18$ | $19.00\pm 0.68$ | $22.50\pm 0.78$ | $7.15\pm 1.82$ | $3.81\pm 0.97$ |
| Drops [] | $73.66\pm 0.29$ | $69.14\pm 0.38$ | $67.15\pm 0.17$ | $58.12\pm 0.26$ | $47.07\pm 0.74$ | $43.42\pm 1.19$ | $50.85\pm 0.49$ | $36.27\pm 1.15$ | $32.43\pm 1.90$ |

## 5 Conclusion

In this paper, we introduced the Automatic Dataset Construction (ADC) pipeline, a novel approach for automating the creation of large-scale datasets with minimal human intervention. By leveraging Large Language Models for detailed class design and automated sample collection, ADC significantly reduces the time, cost, and errors associated with traditional dataset construction methods. The Clothing-ADC dataset, which comprises one million images with rich category hierarchies, demonstrates the effectiveness of ADC in producing high-quality datasets tailored for complex research tasks. Despite its advantages, ADC faces challenges such as label noise and imbalanced data distributions. We addressed these challenges with open-source tools for error detection and robust learning. Our benchmark datasets further facilitate research in these areas, ensuring that ADC remains a valuable tool for advancing machine learning model training.

## Appendix

The appendix is organized as follows:

- $\bullet$
Appendix A includes additional detailed algorithms in the Automatic-Dataset-Construction pipeline.
- $\bullet$
Appendix B contains dataset statistics and more exploratory data analysis of Clothing ADC.
- $\bullet$
Appendix C includes experiment details of our benchmark on label noise detection, label noise learning, and class-imbalanced learning.

### Broader impacts

Our paper introduces significant advancements in dataset construction methodologies, particularly through the development of the Automatic Dataset Construction (ADC) pipeline:

- $\bullet$
Reduction in Human Workload: ADC automates the process of dataset creation, significantly reducing the need for manual annotation and thereby decreasing both the time and costs associated with data curation.
- $\bullet$
Enhanced Data Quality for Research Communities: ADC provides high-quality, tailored datasets with minimal human intervention. This provides researchers with datasets in the fields of label noise detection, label noise learning, and class-imbalanced learning, for exploration as well as fair comparisons.
- $\bullet$
Support for Customized LLM Training: The ability to rapidly generate and refine datasets tailored for specific tasks enhances the training of customized Large Language Models (LLMs), increasing their effectiveness and applicability in specialized applications.

Furthermore, the complementary software developed alongside ADC enhances these impacts:

- $\bullet$
Data Curation and Quality Control: The software aids in curating and cleaning the collected data, ensuring that the datasets are of high quality that could compromise model training.
- $\bullet$
Robust Learning Capabilities: It incorporates methods for robust learning with collected data, addressing challenges such as label noise and class imbalances. This enhances the reliability and accuracy of models trained on ADC-constructed datasets.

Together, ADC and its accompanying software significantly advance the capabilities of machine learning researchers and developers by providing efficient tools for high-quality customized data collection, and robust training.

### Limitations

While ensuring the legal and ethical use of datasets, including compliance with copyright laws and privacy concerns, is critical, our initial focus is on legally regulated and license-friendly data sources available through platforms like Google or Bing. Addressing these ethical considerations is beyond the current scope but remains an essential aspect of dataset usage.

Besides, similar to Traditional-Dataset-Construction (TDC), Automatic-Dataset-Construction (ADC) is also unable to guarantee fully accurate annotations.

## Appendix A Detailed algorithms in the generation of Automatic-Dataset-Construction

### A.1 The algorithm of image data collection in ADC

Figure: Algorithm 2 Image Data Collection in ADC

### A.2 The algorithm of learning-centric curation method in ADC

Figure: Algorithm 3 Learning-centric curation (early-learning memorization behavior)

## Appendix B Dataset statistics in Clothing-ADC

### B.1 Collected Clothing ADC dataset

Our collected Clothing-ADC dataset can be found here: [Google Drive](https://drive.google.com/file/d/1U-NXvHfmUUqL1l5_PspBIcJwXrlxHZe4/view?usp=sharing).

### B.2 Dataset statistics in Clothing-ADC

Our automated dataset creation pipeline is capable of generating numerous designs per attribute, as shown in Table 6. This table provides a detailed list of designs generated by our pipeline, from which we selected a subset to include in our dataset.

**Table 6: The union of attributes across all clothing types in Clothing-ADC dataset.**
| Color | Material | Pattern |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Animal print | Gold | Pastel | Acrylic | Lace | Tulle | Abstract | Camouflage | Fishnet | Leather | Printed | Thongs |
| Beige | Gray | Peach | Alpaca | Leather | Tweed | Abstract Floral | Chalk stripe | Floral | Logo | Quilted | Tie-Dye |
| Black | Green | Pink | Angora | Lightweight | Twill | Animal Print | Check | Floral print | Low rise | Reversible | Tie-dye |
| Blue | Grey | Plum | Bamboo | Linen | Velvet | Animal print | Checkered | Fringe | Mesh | Ribbed | Toile |
| Blush Pink | Heather | Purple | Breathable | Mesh | Viscose | Aran | Chevron | G-strings | Military | Ripples | Trench |
| Bright Red | Ivory | Red | Cashmere | Microfiber | Water-resistant | Argyle | Color block | Galaxy | Mock turtleneck | Satin | Tribal |
| Brown | Khaki | Rich Burgundy | Chambray | Modal | Windproof | Aztec | Colorblock | Garter Stitch | Mosaic | Scales | Tuck stitch |
| Burgundy | Lavender | Royal Blue | Chiffon | Mohair | Wool | Basket check | Cotton | Garter stitch | Moss stitch | Seamless | Tweed |
| Burnt Orange | Light Grey | Rust | Corduroy | Neoprene | acrylic | Basket rib | Cropped | Geometric | Moto | Seed stitch | Twill |
| Champagne | Maroon | Rustic Orange | Cotton | Nylon | bamboo | Basket weave | Damask | Gingham | Nailhead | Shadow stripe | Vintage-inspired |
| Charcoal | Metallic | Sage | Crochet | Organza | cotton | Basketweave | Denim | Glen check | Nehru | Sharkskin | Waterproof |
| Charcoal Grey | Mustard | Silver | Denim | PVC | hemp | Batik | Diagonal grid | Gradient | Nordic | Sherpa | Windowpane |
| Cream | Mustard Yellow | Soft Pink | Down | Polyester | linen | Bikini | Diamond | Graphic | Ombre | Silk |  |
| Cream White | Navy | Striped | Embroidered | Rayon | lycra | Birdseye | Ditsy | Grid | Oversized | Slip Stitch |  |
| Dark Plum | Navy Blue | Tan | Flannel | Reflective | modal | Blazer | Dogtooth | Herringbone | Oxford | Slip stitch |  |
| Deep Blue | Neon | Teal | Fleece | Ripstop | nylon | Bomber | Embossed | High waisted | Paisley | Solid |  |
| Deep Purple | Nude | Turquoise | Fringe | Satin | polyester | Boxer briefs | Embroidered | Honeycomb | Peacoat | Striped |  |
| Earthy Beige | Olive | Vibrant Turquoise | Fur | Silk | rayon | Briefs | Emoji | Houndstooth | Pin Dot | Stripes |  |
| Floral | Olive Green | Warm Brown | Gore Tex | Softshell | silk | Brioche | Entrelac | Ikat | Pinstripe | Studded |  |
| Forest Green | Orange | White | Gore-Tex | Spandex | spandex | Broken rib | Eyelet | Intarsia | Plaid | Suede |  |
| Fuchsia | Pale Yellow | Yellow | Hemp | Suede | tencel | Broken stripe | Fair Isle | Jacquard | Polka Dot | Tartan |  |
|  |  | lilac | Insulated | Synthetic | viscose | Cable | Fibonacci | Knit and Purl | Polka dot | Teddy |  |
|  |  |  | Jersey | Synthetic Blend | wool | Cable knit | Fisherman | Lace | Prince of Wales | Textured |  |
|  |  |  | Knit | Tencel |  |  |  |  |  |  |  |

### B.3 Test set human vote collection

Our automated dataset collection pipeline enabled us to create a large, noisy labeled dataset. We asked annotators to select the best-fitting options from a range of samples, as shown in Figure 5, with each task including at least 4 samples and workers completing 10 tasks per HIT at a cost of $0.15 per task, totaling $150 estimated wage of $2.5-3 per hour, and after further cleaning the label noise, we ended up with 20,000 samples in our test set. To participate, workers had to meet specific requirements, including being Master workers, having a HIT Approval Rate above 85%, and having more than 500 approved HITs, with the distribution of worker behavior shown in Figure 6.

Figure: Figure 5: Collection of Clothing-ADC test set: A filtering task to the worker instead of annotation from scratch.
Refer to caption: extracted/5796241/fig/test_set_collection.jpg

Figure: (a) Distribution of the HITs completed per worker
Refer to caption: extracted/5796241/fig/testset_creationhis_submissions.png

## Appendix C Experiment details

### C.1 Distribution of Human Votes for Label Noise Evaluation

On the annotation page, we presented the image and its original label to the worker and asked if they believed the label was correct (Figure 7). They input their evaluation by clicking one of three buttons. Note that we encouraged workers to categorize acceptable samples as "unsure". The resulting distribution is shown in Table 7. Using a simple majority vote aggregation, we found that the noise rate in our dataset is 22.15%. However, if a higher level of certainty is required for clean labels, we can apply a more stringent aggregation method, considering more samples as mislabeled. In the extreme case where any doubts from any of the three annotators can disqualify a sample, our automatically collected dataset still retains 61.25% of its samples.

For the label noise evaluation task, we utilized a subset of 20,000 samples from the Clothing-ADC dataset, collecting three votes from unique workers for each sample. Each Human Intelligence Task (HIT) included 20 samples and cost $0.05. To participate, workers had to meet the following requirements: (1) be Master workers, (2) have a HIT Approval Rate above 85%, and (3) have more than 500 approved HITs. The total cost for this task was $150, estimated wage of $2.5-3 per hour.

We show the distribution of worker behavior during the noise evaluation task in Figure 8. Figure 8(a) shows the distribution of the amount of HIT completed per worker while neglecting ids with 1-2 submissions. There is a total of 49 unique workers. Figure 8(b) shows the distribution of time spent per HIT.

Figure: Figure 7: Label noise evaluation worker page
Refer to caption: extracted/5796241/fig/label_noise_qc.jpg

Figure: (a) Distribution of the HITs completed per worker
Refer to caption: extracted/5796241/fig/noiseEval_his_submissions.png

**Table 7: Distribution of Human Votes for Label Noise Evaluation: We employed human annotators to evaluate a subset of 20,000 samples from our collected dataset, with each sample receiving three votes from distinct annotators.**
| Human Votes | Percentage |
| --- | --- |
| Yes, Yes, Yes | 61.25% |
| Yes, Yes, Unsure | 6.10% |
| Yes, Yes, No | 10.50% |
| Else | 22.15% |

### C.2 Noisy learning and class imbalance learning benchmark implementation details

Our code refers to zip file in supplementary material.

Figure: Listing 1: How to load data. Line 1 loads the full set of our dataset. Line 2 and Line 3 load the tiny version of our dataset. Line 4 creates the validation set. Line 5 creates the testing set. Line 11 to Line 20 create the data loader.

Figure: Listing 2: The example of the command we use to run the algorithm in one line

Figure: Listing 3: The example of YAML config file

### C.3 Label noise detection

We run four baselines for label noise detection, including CORES [], confident learning [], deep $k$-NN [] and Simi-Feat []. All the experiment is run for one time following [, ].

The experiment platform we run is a 128-core AMD EPYC 7742 Processor CPU and the memory is 128GB. The GPU we use is a single NVIDIA A100 (80GB) GPU. For the dataset, we used human annotators to evaluate whether the sample has clean or noisy label as mentioned in Appendix C.1. We aggressively eliminates human uncertainty factors and only consider the case with unanimous agreement as a clean sample, and everything else as noisy samples. The backbone model we use is ResNet-50 [].

For all the baselines, the parameters we use are the same as the original paper except the data loader. We skip the label corruption and use the default value from the original repository.
For CORES, the cores loss whose value is smaller than 0 is regarded as the noisy sample.
For confidence learning, we use the repository(^1^11https://github.com/cleanlab/cleanlab) from the clean lab and the default hyper-parameter.
For deep $k$-NN, the $k$ we set is 100.
For SimiFeat, we set $k$ as 10 and the feature extractor is CLIP.

### C.4 Label noise learning

The platform we use is the same as label noise detection. The backbone model we use is ResNet-50 []. For the full dataset, we run the experiment for 1 time. For the tiny dataset, we run the experiments for 3 times. The tiny dataset is sampled from the full set whose size is 50. The base learning rate we use is 0.01. The base number of epochs is 20. The hyper-parameters for each baseline method are as follows.
For backward and forward correction, we train the model using cross-entropy (CE) loss for the first 10 epochs. We estimate the transition matrix every epoch from the 10th to the 20th epoch. For the positive and negative label smoothing, the smoothed labels are used at the 10th epoch. The smooth rates of the positive and negative are 0.6 and -0.2. Similarly, for peer loss, we train the model using CE loss for the first 10 epochs. Then, we apply peer loss for the rest 10 epochs and the learning rate we use for these 10 epochs is 1e-6. The hyper-parameters for $f$-div is the same as those of peer loss. For divide-mix, we use the default hyper-parameters in the original paper. For Jocor, the hyper-parameters we use is as follows. The learning rate is 0.0001. $\lambda$ is 0.3. The epoch when the decay starts is 5. The hyper-parameters of co-teaching is similar to Jocor. For logitclip, $\tau$ is 1.5. For taylorCE, the hyper-parameter is the same as the original paper.

### C.5 Class-imbalanced learning

The platform we use is the same as label noise detection. The backbone model we use is ResNet-50 []. For different imbalance ratio ($\rho=10,50,100$). The class distribution is shown in Table 8. For all the methods, the base learning rate is 0.0001 and the batch size is 448. The dataset we use is not full dataset because we want to disentangle the noisy label and class imbalance learning. We use Docta and a pre-trained model trained with cross-entropy to filter the data whose prediction confidence is low. Due to the memorization effect, we fine-tune the model for 2 epochs to filter the data. We remove 45.15% data in total where Docta removes 26.36% while CE removes 25.00% with a overlap of 6.20%. Thus, the datset we use for class-imbalance learning is 54.85% of the full dataset.

**Table 8: The class distribution for different imbalance ratio**
| imbalance ratio ($\rho$) | Class Distribution | Total Number |
| --- | --- | --- |
| 10 | [39297, 31875, 25854, 20971, 17010, 13797, 11191, 9078, 7363, 5972, 4844, 3929] | 191181 |
| 20 | [39297, 27536, 19295, 13520, 9474, 6638, 4652, 3259, 2284, 1600, 1121, 785] | 129461 |
| 100 | [39297, 25854, 17010, 11191, 7363, 4844, 3187, 2097, 1379, 907, 597, 392] | 114118 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Please discuss the applicability of the proposed methodology for other domains such as text. How will it translate - some components (cleaning) are straightforward, others not so much (web scraping or labeling)? Please clarify if it doesn't apply. 
2. It will be helpful to have a more elaborate discussion on synthetic datasets. The paper already includes benefits of ADC over TDC. ADC should be clearly contrasted against synthetic data. Can some of the issues introduced by hallucination be recitified in cleaning stage?
3. Please include statistics for class distribution - this will provide a better idea of imbalance as well.
4. The following sentence is a bit unclear "For applications where some label noise can be tolerated, existing data curation software capable of identifying and filtering out irrelevant images, such as Docta, CleanLab , and Snorkel 1, etc."
soundness: 3
strengths: 1. The paper is well written and easy to follow.
2. The paper tackles an important problem of dataset creation - the proposed approach (ADC) can do it in cost effective manner. Table 7 provides a comparison against existing label noise datasets including Cifar 10 N / H and Cifar to demonstrate the effectiveness of ADC.
3. The paper presents the Clothing-ADC dataset with 1M+ samples with 12K classes. 
4. The paper also presents a subset of Clothing-ADC CLT which is suitable for class imbalance learning. Various baselines (Drops, Bal-softmax, Logit-Adjust) are also reported for this dataset. A subset of 20K samples is also proposed for label noise detection.
5. It is important to discuss the biases (tail or infrequent classes) and noise (wrongly labeled samples) introduced due to the web data - the paper does so clearly.
6. The code and hyper-parameter details are clearly specified.
summary: The paper tackles the problem of automatic dataset creation. It proposes the ADC pipeline for the same which requires minimal human overhead. The proposal is to first decide the attributes (using a LLM) and then crawl data using attribute as filters (using Google and BING APIs). This is in contrast to existing approaches that label instances given a list of classes. Finally the instances are cleaned to filter noisy labeled samples. 

The paper proposes Clothing-ADC dataset with 1M+ samples. Moreover, the paper also discusses the challenges associated with dataset construction including label errors, noisy labels and class imbalance. It then presents a solution (based on existing approaches for noise detection) to tackle these challenges to clean or improve reliability of the constructed dataset.
weaknesses: 1. The ADC methodology is general; however, the paper applies it only to image data (Clothing-ADC dataset). A brief discussion can be included for other domains (such as text), if it applies. 
2. The approach is specifically designed for cases when data is fetched as part of process. It may not apply to the cases where data is available in some form such as unlabelled corpus.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could the authors clarify the rationale behind designing specific benchmark datasets for label noise and class imbalance in the context of an automatic dataset construction pipeline? How do these benchmarks align with the overall goal of automation?

2. Given that the ADC pipeline relies on LLMs for dataset construction, what measures are in place to ensure the accuracy and relevance of data collected, especially in cases where human annotation is minimized?
soundness: 2
strengths: The paper is well-organized and clearly written, easy to follow.
summary: The paper presents the Automatic Dataset Construction (ADC) pipeline, which automates the dataset creation process by leveraging large language models (LLMs) for sample collection, reducing manual annotation efforts, and enhancing efficiency in dataset generation.
weaknesses: 1. The paper’s purpose feels somewhat misaligned with practical needs. If the goal is truly Automatic Dataset Construction, the intentional design of three benchmark datasets focusing on label noise detection, label noise learning, and class-imbalance learning seems contradictory. The intent behind this approach isn’t entirely clear to me.

2. The novelty is limited, as the methodology primarily relies on leveraging LLMs to construct the dataset, which feels more like engineering work than a novel research contribution.

3. Lack of clarity regarding how the ADC pipeline addresses potential copyright and ethical concerns associated with using LLMs for data collection.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How well will the human-in-the-loop step would scale as the dataset size gets larger, or if there is high levels of noise? Have the authors considered any ranking mechanism to prioritize samples for human review, or as another automatic filtering technique?
2. The current pipeline only checks for low-quality data in the form of noisy labels and imbalanced classes, have the authors considered how ADC could also address other data issues such as outliers, duplicates, and other low quality data (for images, that could include blurry images, odd aspect ratios etc)?
soundness: 3
strengths: 1. New methodology proposed that will enable efficient dataset construction across many domains, making it adaptable for broad future applications
2. Creation of Clothing-ADC dataset which supports future work on label noise and class imbalance
3. The paper acknowledges challenges that ADC encounters when it comes to ensuring data quality, and the authors present and benchmark a comprehensive set of tools that could help to address these limitations, which helps to make the approach more viable
summary: This paper introduces Automatic Dataset Construction (ADC) a new methodology to create large-scale datasets with reduced costs and improved efficiency and accuracy. This approach leverages large language models (LLMs) for class design and sample collection, then uses various existing label error detection and robust learning techniques to enhance the quality of the dataset. The authors introduced a new dataset curated using the previously described methods, Clothing-ADC. They also developed versions of the dataset to facilitate benchmarks in label noise detection, label noise learning, and class-imbalanced learning, which they also presented comprehensive results and evaluations for.
weaknesses: 1. Lack of novelty in data quality methods. The paper and pipeline relies heavily on existing methods to handle label noise and class imbalance, and does not introduce any new methodologies. It mainly offers a procedural contribution rather than advancing new techniques for improving label noise or class imbalance handling
2. The paper presents many algorithms to identify label noise but is lacking detailed analysis of the impact of each algorithm on the resulting dataset and downstream model performance
3. Related to the point above, the paper does not provide quality metrics to evaluate the quality of the dataset after before/after using the ADC pipeline to clean the data, this makes it hard to assess the effectiveness of the automated curation process and ADC’s ability to produce high quality data

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please the Weaknesses above.
soundness: 3
strengths: 1. The proposed Automatic Dataset Construction (ADC) leverages LLMs for the detailed class design and code generation to collect relevant samples via search engines, reducing the need for manual annotation and speeding up the data generation process.

2.  The authors explore several challenges observed in real-world dataset construction, including detecting label errors, learning with noisy labels, and class-imbalanced learning.
summary: In this paper, the authors proposed propose Automatic Dataset Construction (ADC), a methodology that automates dataset creation with negligible cost and high efficiency. ADCleverages LLMs for the detailed class design and code generation to collect relevant samples via search engines. The authors design three benchmark datasets focused on label noise detection, label noise learning, and class-imbalanced learning.
weaknesses: Could the authors please address the following questions:

1. In Step 2 of Fig. 1, the explanation for "image search engine" states, "Labeling, ADC reduces human workload by flipping the data collection process, using targets to search for samples." What is the conceptual meaning of this part?

2. In Step 3 of Fig. 1, what is the insight behind using "filter" instead of "relabeling"?

Meta-review: This paper was reviewed by four experts in the field and received 8, 5, 5, 6 as the final ratings. The reviewers agreed that the paper tackles an important problem of dataset creation, it is well-written and easy to follow, and the hyper-parameter details are clearly specified. The created Clothing-ADC dataset, with more than 1 million samples and 12k classes, will be useful to the community. 

Reviewers YU7i and kh2s raised a concern about the novelty of the method, as it mostly uses existing methods for dataset creation, handling label noise and class imbalance. While the authors have explained in the rebuttal that this may be appropriate for the "datasets and benchmarks" track of the conference, the AC believes that the novelty is still limited, given the standards of ICLR. Moreover, a thorough comparison against other automated dataset creation pipelines is necessary to appropriately understand the usefulness and merit of the proposed ADC methodology. 

Reviewer YU7i also mentioned that the connection between the benchmarks (focused on label noise detection and class imbalance learning) and the goal of automated dataset construction is unclear. The authors have explained in the rebuttal that the benchmarks have been designed to validate the effectiveness of the ADC software solutions (the label noise and class imbalance learning benchmarks are used to demonstrate the effectiveness when working with imperfect training data). However, the class-imbalance learning baselines and the label noise detection baselines studied in the paper are all existing methods and none of them have been proposed by the authors. Thus, the need to validate their effectiveness remains unclear and disconnected from the main objective of the paper. A question was also raised about other kinds of data issues such as outliers, duplicates, blurry images, which was not addressed convincingly by the authors in the rebuttal. 

A few ethical concerns were also raised, such as the filtering of inappropriate or unethical images and addressing copyright concerns. These were not addressed convincingly by the authors and their reliance on the "safe search" option of search engines is insufficient to safeguard against insufficient or unethical data. 

We appreciate the authors' efforts in meticulously responding to each reviewer's comments. However, in light of the above discussions, we conclude that the paper may not be ready for an ICLR publication in its current form. While the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.

Decision: Reject