SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mark Mazumder<sup>1</sup>, Colby Banbury<sup>1</sup>, Xiaozhe Yao<sup>2</sup>, Bojan Karlas<sup>2</sup>, William Gaviria Rojas<sup>3</sup>, Sudnya Diamos<sup>3</sup>, Greg Diamos<sup>4</sup>, Lynn He<sup>5</sup>, Alicia Parrish<sup>8</sup>, Hannah Rose Kirk<sup>17</sup>, Jessica Quaye<sup>1</sup>, Charvi Rastogi<sup>11</sup>, Douwe Kiela<sup>9,21</sup>, David Jurado<sup>6,20</sup>, David Kanter<sup>6</sup>, Rafael Mosquera<sup>6,20</sup>, Juan Ciro<sup>6,20</sup>, Lora Aroyo<sup>8</sup>, Bilge Acun<sup>7</sup>, Lingjiao Chen<sup>9</sup>, Mehul Smriti Raje<sup>3</sup>, Max Bartolo<sup>16,19</sup>, Sabri Eyuboglu<sup>9</sup>, Amirata Ghorbani<sup>9</sup>, Emmett Goodman<sup>9</sup>, Oana Inel<sup>18</sup>, Tariq Kane<sup>3,8</sup>, Christine R. Kirkpatrick<sup>10</sup>, Tzu-Sheng Kuo<sup>11</sup>, Jonas Mueller<sup>12</sup>, Tristan Thrush<sup>9</sup>, Joaquin Vanschoren<sup>13</sup>, Margaret Warren<sup>14</sup>, Adina Williams<sup>7</sup>, Serena Yeung<sup>9</sup>, Newsha Ardalani<sup>7</sup>, Praveen Paritosh<sup>6</sup>, Ce Zhang<sup>2</sup>, James Zou<sup>9</sup>, Carole-Jean Wu<sup>7</sup>, Cody Coleman<sup>3</sup>, Andrew Ng<sup>4,5,9</sup>, Peter Mattson<sup>8</sup>, and Vijay Janapa Reddi<sup>1</sup>
$^{1}$ Harvard University,  $^{2}$ ETH Zurich,  $^{3}$ Coactive.AI,  $^{4}$ Landing AI,  $^{5}$ DeepLearning.AI,  $^{6}$ MLCommons,  $^{7}$ Meta,  $^{8}$ Google,  $^{9}$ Stanford University,  $^{10}$ San Diego Supercomputer Center, UC San Diego,  $^{11}$ Carnegie Mellon University,  $^{12}$ Cleanlab,  $^{13}$ Eindhoven University of Technology,  $^{14}$ Institute for Human and Machine Cognition,  $^{15}$ Kaggle,  $^{16}$ Cohere,  $^{17}$ University of Oxford,  $^{18}$ University of Zurich,  $^{19}$ University College London,  $^{20}$ Factored,  $^{21}$ Contextual AI
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems. Neglecting the fundamental importance of data has given rise to inaccuracy, bias, and fragility in real-world applications, and research is hindered by saturation across existing dataset benchmarks. In response, we present DataPerf, a community-led benchmark suite for evaluating ML datasets and data-centric algorithms. We aim to foster innovation in data-centric AI through competition, comparability, and reproducibility. We enable the ML community to iterate on datasets, instead of just architectures, and we provide an open, online platform with multiple rounds of challenges to support this iterative development. The first iteration of DataPerf contains five benchmarks covering a wide spectrum of data-centric techniques, tasks, and modalities in vision, speech, acquisition, debugging, and diffusion prompting, and we support hosting new contributed benchmarks from the community. The benchmarks, online evaluation platform, and baseline implementations are open source, and the MLCommons Association will maintain DataPerf to ensure long-term benefits to academia and industry.
Machine learning research has overwhelmingly focused on improving models rather than on improving datasets. Large public datasets such as ImageNet [14], Freebase [7], Switchboard [22], and SQuAD [44] serve as compasses for benchmarking model performance. Consequently, researchers eagerly adopt the largest existing dataset without fully considering its breadth, difficulty and fidelity to the underlying problem. Critically, better data quality [2] is increasingly necessary to improve generalization, avoid bias, and aid safety in data cascades [48]. Without high-quality training data models can exhibit performance discrepancies leading to reduced accuracy and persistent fairness
Figure 1: Typical benchmarks are model-centric, and therefore focus on the model design and training stages of the ML pipeline (shown in orange). However, to develop high-quality ML applications, users often employ a collection of data-centric operations to improve data quality and repeated data-centric iterations to refine these operations. DataPerf aims to benchmark all major stages of such a data-centric pipeline (shown in green) to improve ML data quality.
issues [9, 15, 37] once they leave the lab to enter service. In conventional model-centric ML, the term benchmark often means a standard, fixed dataset for model accuracy comparisons and performance measurements. While this paradigm has been useful for advancing model design, these benchmarks are now saturating (attaining perfect or above "human-level" performance) [26]. This raises two questions: First, is ML research making real progress on the underlying capabilities, or is it just overfitting to existing benchmark datasets or suffering from data artifacts? A growing body of literature explores the evidence supporting benchmark limitations [57, 24, 43, 53, 47, 5, 21, 55]. Second, how should benchmarks evolve to push the frontier of ML research?
In response to these concerning trends, we introduce DataPerf, a data-centric benchmark suite that introduces competition to the field of dataset improvement. We survey a suite of complex data-centric development pipelines across multiple ML domains and isolate a subset of concrete tasks that we believe are representative of current bottlenecks, as illustrated in Figure 1. We freeze model architectures, training hyperparameters, and task metrics to compare solutions strictly via relative improvements from changes to the datasets themselves.
- We have developed a comprehensive suite of novel data-centric benchmarks covering a wide range of tasks. These tasks encompass training set selection for speech and vision, data cleaning and debugging, data acquisition, and diffusion model prompting.
- Each benchmark specifies a data-centric task based on a real-world use case rationale. We provide rules for submissions, along with evaluation scripts, and a baseline submission for each benchmark task.
- We provide an extensible and open-source platform for hosting data-centric benchmarks, allowing other organizations and researchers to propose new benchmarks for inclusion in the DataPerf suite, and to host data challenges themselves.
Critically, DataPerf is not a one-off competition. We have established the DataPerf Working Group, which operates under the MLCommons Association. This working group is responsible for the ongoing maintenance of the benchmarks and platform, as well as for fostering the development of data-centric research and methodologies in both academic and industrial domains. The aim is to ensure the long-term sustainability and growth of DataPerf beyond a single competition.
The remainder of the paper is organized as follows. In Section 2.1, we review lessons learned from an exploratory data-centric challenge. Section 2.2 details the hosting platform we developed in response and Section 2.3 presents the DataPerf suite of five novel benchmarks and challenges. We conclude with a survey of related efforts (Section 3) and future directions (Section 5).
2 DataPerf Benchmarking Suite
We describe the initial challenge which inspired the suite of DataPerf benchmarks and identified which features are needed for hosting data-centric challenges online. We then describe the platform that enables flexible data-centric benchmarking at scale. Finally, we share the initial DataPerf benchmark definitions in vision, speech, acquisition, debugging, and text-to-image prompting.
2.1 The Data-Centric AI Challenge
The DataPerf effort began with an early benchmark which served to validate feasibility and provide real-world insights into the concept of dataset benchmarking. In traditional ML challenges, contestants must train a high-accuracy model given a fixed dataset. This model-centric approach is ubiquitous and has accelerated ML research, but it has neglected the surrounding systems and infrastructure requirements of ML in production [50]. To draw more attention to other areas of the ML pipeline, we created the Data-Centric AI (DCAI) competition [39], inviting competitors to focus on optimizing accuracy by improving a dataset given a fixed model architecture, thus flipping the conventional challenge format of submitting different models which are evaluated on a fixed dataset. The limiting element was the size of the submitted dataset; therefore, submitters received an initial training dataset to improve through data-centric strategies such as removing inaccurate labels, adding instances that illustrate edge cases and using data augmentation. The competition, inspired by MNIST, focuses on classification of Roman-numeral digits. Just by iterating on the dataset, participants increased the baseline accuracy from  $64.4\%$  to  $85.8\%$ ; human-level performance (HLP) was  $90.2\%$ . We learned several lessons from the 2,500 submissions and applied them to DataPerf:
1. Common data pipelines. Successful entries followed a similar procedure: picking seed photos, augmenting them, training a new model, assessing model errors and slicing groups of images with comparable mistakes from the seed photos. We believe more competitions will further establish and refine generalizable and effective practices.
2. Automated methods won. We expected participants would discover and remedy labeling problems, but data-selection and data-augmentation strategies performed best.
3. Novel dataset optimizations. Examples of successful tactics include automated methods for recognizing noisy images and labels, identifying unlabeled images, defining explicit labeling rules for confusing images, correcting class imbalance, and selecting and enhancing images from the long tail of classes. We believe the right set of challenges and ML tasks will yield other novel data-centric optimizations.
4. New methods emerged. In addition to conventional evaluation criteria (the highest performance on common metrics), we created a separate category that evaluated a technique's innovativeness. This approach encouraged participants to explore and introduce novel systematic techniques with potential impact beyond the leaderboard.
5. New supporting infrastructure is necessary. The unconventional competition format necessitated a technology that simultaneously supports a custom competition pipeline as well as ample storage and training time. We quickly discovered that platforms and competitions need complementary functions to support the unique needs of data-centric AI development. Moreover, the competition was computationally expensive. Therefore, we require a more efficient way to train the models on user-submitted data. Computational power, memory and bandwidth are all major limitations.
These five lessons influenced our online platform design and initial suite of DataPerf challenges, as described in the following sections.
2.2 Evaluation Platform
DataPerf provides an online platform where challenge participants can submit their solutions for evaluation, and a working group which invites members in academia and industry to propose new data-centric benchmarks for inclusion in the DataPerf suite. The DataPerf benchmarks, evaluation tools, leaderboards, and documentation are hosted in an online platform called Dynabench[26],
<https://dynabench.org/>
which allows challenge participants to submit, evaluate, and compare solutions for all data-centric benchmarks defined in Section 2.3. The DataPerf benchmarks and the Dynabench platform are open-source, and are hosted and maintained by the MLCommons Association $^{2}$ , a nonprofit organization supported by more than 50 member companies and academics, ensuring long-term availability and benefit to the community.
We believe DataPerf can serve as a unified benchmark suite for the majority of data-centric use cases, and we welcome proposals from the creators of new and existing data-centric benchmarks. Our five current benchmarks are also intended to serve as representative examples for future authors to host their own challenges on DataPerf, with customized modular submission pipelines for different data modalities and submission artifact types. DataPerf introduces three key extensions to the Dynabench codebase to support data-centric benchmarks: (1) We add support for a wide variety of submission artifacts, such as training subsets, priority values/orderings, and purchase strategies. Users can also submit fully containerized systems as artifacts, such as in the debugging challenge. (2) To support a diverse set of evaluation algorithms and scoring metrics, we develop modular software adaptors to allow for running custom benchmark evaluation tools and displaying or querying scores in Dynabench's online leaderboards. (3) DataPerf utilizes serverless [4] deployment which dynamically scales resources based on demand, ensuring optimal performance and efficient resource allocation, and allowing the platform to automatically scale with the growth of the benchmark suite and the number of participants. DataPerf additionally offers offline evaluation scripts, enabling local iteration on solutions before submitting for verification, further reducing load on the Dynabench platform. These improvements to Dynabench ensure DataPerf can accommodate a large suite of community-contributed data-centric challenges in the future.
2.3 Challenges, Benchmarks, and Leaderboards
DataPerf uses leaderboards and challenges to encourage constructive competition and inspire advances in building and optimizing datasets. In this section, we clarify DataPerf's terminology. A leaderboard is a public summary of benchmark results; it helps to quickly identify state-of-the-art approaches. A challenge is a public contest to achieve the best result on a leaderboard in a fixed timeframe. Challenges motivate rapid progress through recognition and awards. Our leaderboards and challenges are hosted on the online platform Dynabench (Section 2.2) developed and supported by MLCommons. Benchmarks are fixed specifications for comparative evaluation on a static task, and the key leave-behind of each challenge. MLCommons will provide long-term support for each benchmark through leaderboards which remain open for submission and comparison once a challenge concludes. Each challenge also provides a baseline implementation to set a minimum bar for each leaderboard metric and to discourage uninformative or random submissions.
DataPerf's initial suite consists of tasks in training set selection for speech and vision, data cleaning and debugging, data acquisition, and generative model prompting. Figure 1 depicts underserved components in benchmarking machine learning pipelines, and these five tasks were selected by the DataPerf working group among the initial proposals for challenges in order to cover as many of these components as possible while also exercising the infrastructure requirements for our online platform. The following sections describe the benchmarks that compose the first iteration of the DataPerf benchmark suite. Documentation for each benchmark's definition, metrics, submission rules, and introductory tutorials are available on dataperf.org and reproduced in our Appendix, and our open-source baseline implementations are available at https://github.com/MLCommons/dataperf.
2.3.1 Selection for Speech
DataPerf includes a dataset-selection-algorithm challenge with an emphasis on low-resource speech. The objective of the speech selection task is to develop a selection algorithm that chooses the most effective training samples from a large and noisy multilingual corpus of spoken words, expanding sample quality estimation techniques to low-resource language settings. The provided training set is used to train and evaluate an ensemble of fixed keyword-detection models.
Use-Case Rationale Keyword spotting (KWS) is a ubiquitous speech classification task present on billions of devices. A KWS model detects a limited vocabulary of spoken words. Production
$^{2}$ https://www.mlcommons.org/
Figure 2: System design and component ownership for the speech selection benchmark.
examples include the Wakeword interfaces for Google Voice Assistant, Siri and Alexa. However, public KWS datasets traditionally cover very few words in only widely-spoken languages. In contrast, the Multilingual Spoken Words Corpus [35] (MSWC), is a large dataset of over 340,000 spoken words in 50 languages (collectively, these languages represent more than five billion people). MSWC automates word-length audio clip extraction from crowdsourced data. Due to errors in the generation process and source data, some samples are incorrect. For instance, they may miss part of the target sample (e.g., "weather-" instead of "weather") or may contain part of an adjacent word (e.g., "time to" instead of "time"). This benchmark focuses on estimating the quality of each automatically-generated sample in KWS training pipelines intended for low-resource languages. Additionally, this benchmark establishes the DataPerf platform's capabilities for hosting speech challenges in multiple languages.
Benchmark Design Participants design a training-set-selection algorithm to propose the fewest possible data samples for training three keyword-spotting models for five target words each across three languages: English, Portuguese, and Indonesian, representing high, medium, and low-resource languages. The benchmark evaluates the algorithm on the mean  $F_{1}$  score of each evaluation set (additional details in Appendix A.3). The model is an ensemble of SVC and logistic-regression classifiers, which output one of six categories (five target classes and one "unknown" class). The inputs to the classifier are 1,024-dimensional vectors of embedding representations from a pretrained keyword feature extractor [34]. Participants may only specify which training samples are used by the model; all other configuration parameters are fixed, thereby emphasizing the importance of selecting the most informative samples. For each language there are separate leaderboards for submissions with  $\leq 25$  samples or  $\leq 60$  samples, evaluating the algorithm's sensitivity to the training set size.
Participants are given a tutorial baseline which uses crossfold validation in a Google Colab notebook and an offline copy of the evaluation pipeline, for ease of setup and and rapid experimentation. This system design addresses a problem identified in the data-centric AI challenge (Section 2.1) - enabling offline development reduces the computational requirements for online evaluation, though participants must agree to challenge rules on not inspecting the evaluation set. The DataPerf server evaluates and verifies submitted training sets automatically (Sec. 2.2) for inclusion in the live leaderboard. Figure 2 illustrates the speech-selection benchmark workflow.
Baseline Results We provide two baseline implementations, nested cross-fold selection and a data-cleaning approach using the Cleanlab framework [40]. The cross-fold selection method uses nested cross-validation where the outer loop selects different subsets of the target samples and the inner loop selects different subsets of the nontarget samples, and the best performing subsets are reported back as the selected training set. The Cleanlab method rejects outliers using out-of-sample predicted probability estimates for each candidate sample (also computed via cross-validated models). All baseline scores are averaged across 10 random seeds.
Table 1: Baseline results (macro  $\mathrm{F}_{1}$  scores) for the Selection for Speech challenge.
English	Portuguese	Indonesian
Training set size	25	60	25	60	25	60
Nested cross-fold	0.32	0.41	0.42	0.52	0.36	0.42
Cleanlab	0.49	0.49	0.47	0.57	0.37	0.43
2.3.2 Selection for Vision
DataPerf includes a data selection algorithm challenge with a vision-centric focus. The objective of this task is to develop a data selection algorithm that chooses the most effective training samples from a large candidate pool of images. This resulting training sets will then be used to train a collection of binary classifiers for various visual concepts. The benchmark evaluates the algorithm on the basis of the resulting models' classification performance on the evaluation set.
Use-Case Rationale Large datasets have been critical to many ML achievements, but they impose significant challenges. Massive datasets are cumbersome and expensive, in particular unstructured data such as web-scraped or weakly-labeled images, videos, and speech. Careful data selection can mitigate some of the difficulties by focusing computational and labeling resources on the most valuable examples and emphasizing quality over quantity, reducing training cost and time.
The vision-selection-algorithm benchmark evaluates binary classification of visual concepts (e.g., "monster truck" or "jean jacket") in unlabeled images. Familiar production examples of similar models include automatic labeling services by Amazon Rekognition, Google Cloud Vision API and Azure Cognitive Services. Successful approaches to this challenge will enable image classification of long-tail concepts where discovery of high-value data is critical, and advances the democratization of computer vision [20]. This benchmark demonstrates DataPerf's support for challenges with unlabeled image data and is a template for future benchmarks that target automatic labeling.
Benchmark Design The task is to design a data-selection strategy that chooses the best training examples from a large pool of training images. We evaluate submissions on their ability to algorithmically propose a subset of the Open Images Dataset V6 training set [29] that maximizes the mean F1-score over a set of fixed concepts ("cupcake," "hawk" and "sushi"). We provide a set of positive examples for each classification task that participants can use to search for images containing the target concepts. Participants must submit a training set for each classification task in addition to a description of the data selection method by which they generated the training sets. The challenge platform (Sec. 2.2) automates evaluation of submissions.
Baseline Results We provide three baseline results, namely, farthest point sampling, pseudo-label generation, and modified uncertainty sampling. Farthest point sampling selects negative examples by attempting to sample the feature search space through iterative maximum  $l_{2}$  distances, afterwards returning the best coreset under nested cross-validation. Pseudo label generation trains multiple neural networks and classical models on a subset of data to classify the remainder of points and uses the best-performing model for coreset proposal under multiple sampling experiments. Modified uncertainty sampling trains a binary classifier on noisy positive labels from OpenImages and uses this classifier to assign positive and negative image pools, with the coreset randomly sampled from both pools. For each baseline,  $F_{1}$  scores on the three test concepts are provided in Table 2.
Table 2: Baseline results  $\left( {\mathrm{F}}_{1}\right.$  scores) for the Selection for Vision challenge.
Cupcake	Hawk	Sushi	Mean F1-score
Farthest point sampling	0.75	0.87	0.82	0.81
Pseudo label generation	0.70	0.86	0.81	0.79
Modified uncertainty sampling	0.71	0.83	0.80	0.78
2.3.3 Debugging for Vision
The debugging challenge is to detect candidate data errors in the training set that cause a model to have inferior quality. The aim is to assist a user in prioritizing which samples to inspect, correct, and clean. A debugging method's purpose is to identify the most detrimental data points from a potentially noisy training set. After inspecting and correcting the selected data points, the cleaned dataset is used to train a new classification model. Evaluation is based on the number of data points the debugging approach must correct to attain a certain accuracy.