SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Use-Case Rationale Datasets are rapidly growing in size. For instance, Open Images V6 has 59 million image-level labels. Such datasets are annotated either manually or using ML. Unfortunately, noise is unavoidable and can originate from both human annotators and algorithms. Models trained on noisy annotations suffer in accuracy and carry risks of bias and unfairness. Dataset cleaning is a common approach to dealing with noisy labels. However, it is a costly and time-consuming process that typically involves human review. Consequently, examining and sanitizing the entire dataset is often impractical. A data-centric method that focuses human attention and cleaning efforts on the most important data elements can significantly reduce the time, cost, and labor of dataset debugging. This challenge demonstrates the DataPerf platform's ability to simulate human-in-the-loop data-centric tasks, in this case label cleaning, while remaining scalable.
Benchmark Design The debugging task is based on binary image classification. For each activity, participants receive a noisy training set (i.e., some labels are inaccurate) and a validation set with correct labels. They must provide a debugging approach that assigns a priority value (harmfulness) to each training set item. After each trial, all training data will have been examined and rectified. Each time a new item is examined, a classification model is trained on the clean dataset, and the test accuracy on a hidden test set is computed. Then a score is returned.
The image sets are from the Open Images Dataset [29], with two important considerations: (1) The number of data points should be sufficient to permit random selection of samples for the training, validation and test sets. (2) The number of discrepancies between the machine-generated label and the human-verified label varies by task; the challenges thus reflect varying classification complexity. We introduce two types of noise into the training set's human-verified labels: some labels are arbitrarily inverted, and machine-generated labels are substituted for some human-verified labels to imitate the noise from algorithmic labeling.
We use a 2,048-dimensional vector of embedding representations extracted from a pretrained ResNet50 model [32] as the classifier's input data. Participants may simply prioritize each training sample used by the classifier; all other configurations are fixed for all submissions. By precomputing all embeddings, participants are encouraged to propose data-centric debugging methods for arbitrary features rather than approaches specific to the image domain. This also removes the need for GPU acceleration during submission evaluation.
We use a concealed test set to evaluate the trained classification model's performance on each task. Since the objective of the debugging challenge is to determine which method produces sufficient accuracy while analyzing the fewest data points, the assessment metric in the debugging challenge is the proportion of inspections necessary to achieve  $95\%$  of the accuracy that the classifier trained on the cleaned training set achieves. We verify submissions by incrementally cleaning the data and training a model on each step. Each submission contains a list of indices in the order that the submitter wishes to clean. We incrementally prepare a new dataset for each cleaned sample. For instance, assuming the submission is [5,4,3,2,1], we will prepare 5 datasets that are [5-cleaned, 4,3,2,1], [5-cleaned, 4-cleaned, 3, 2, 1], and so forth. We then train a XGBoost classifier on each dataset, and report back the step at which the accuracy is high enough ( $>95\%$ ) on the test dataset.
Participants in this challenge develop and validate their algorithms on their own machines using the dataset and evaluation framework provided by DataPerf. Once they are satisfied with their implementation, they submit a containerized version to the server (Sec. 2.2). The server then reruns the uploaded implementation on several hidden tasks and posts the average score to a leaderboard.
Baseline Results The benchmark system provides three baseline implementations: consecutive, random and DataScope [25], which achieve the score of 53.50, 51.75 and 15.54 respectively. In other words, DataScope needs to fix  $15.54\%$  of data samples to achieve the threshold, consecutive needs  $53.50\%$  and random needs to fix  $51.75\%$ . DataScope is a fast approximation for Shapley values [31] for importance estimates of each sample included and the effect of noise. As Shapley values require calculating the payoff of every subset  $(\mathrm{O}(2^{\mathsf{N}})$  evaluations), approximation techniques such as DataScope are necessitated.
2.3.4 Data Acquisition
The data acquisition challenge explores which dataset or combination of datasets to purchase in a multi-source data marketplace for specific ML tasks.
Figure 3: Data acquisition benchmark design. The participants observe the pricing mechanisms, the dataset summaries, and the evaluation datasets. They then need to develop and submit the data acquisition strategies. The evaluation is executed automatically on the DataPerf server.
Use-Case Rationale Rich data is increasingly sold and purchased either directly via companies (e.g., Twitter [54] and Bloomberg [6]) or data marketplaces (e.g., Amazon AWS Data Exchange [1], Databricks Marketplace [13], and TAUS Data Marketplace [52]) to train a high-quality ML model customized for specific applications. Those datasets are necessary often because the datasets (i) cover underrepresented populations, (ii) offer high-quality annotations, and (iii) exhibit easy-to-use formats. On the other hand, the datasets are also expensive due to the tremendous efforts spent to curate and clean data samples. Content opacity is therefore ubiquitous: data sellers usually are disinclined to release the full content of their datasets to the buyers. This renders it challenging for the data users to decide whether a dataset is useful for the downstream ML tasks. Based on our conversations with practitioners, existing data acquisition methods for ML are ad-hoc: one has to manually identify data sellers, articulate their needs, estimate the data utilities, and then purchase them. It is also iterative in nature: the datasets may show limited improvements on a downstream ML task after being purchased, and then one has to search for a new dataset again. With this in mind, the goal of this challenge is to mitigate a data buyer's burden by automating and optimizing the data acquisition strategies. This challenge demonstrates the platform's ability to handle datavaluations and demonstrates a unique metric based on a pricing function and a budget, which is a useful template for future challenges that wish to capture the nuance of resource expenditure.
Benchmark Design Participants in this challenge must submit a data acquisition strategy. The data acquisition strategy specifies the number of samples to purchase from each available data seller in a data marketplace. Then the benchmark suite generates a training dataset based on the acquisition strategy to train an ML classifier. To mimic data acquisition in a real-world data marketplace, participants do not have access to sellers' data. Instead, the participants are offered (1) a few samples  $(= 5)$  from each data seller, (2) summary statistics about each dataset, (3) the pricing functions that quantify how much to pay when a particular number of samples is purchased from one seller, and (4) a budget constraint. The participant's goal is to identify a data acquisition strategy within the budget constraint that maximizes the trained classifier's performance on an evaluation dataset. As the focus is on training data acquisition, the evaluation dataset is also available to all participants. The overall system design can be found in Figure 3.
Table 3: We measure three baselines' performance on all five data market instances. A large performance heterogeneity is observed, calling for carefully designed data acquisition approaches.
Market Instance	0	1	2	3	4
Baselines Performance	UNIFORM	0.732	0.757	0.771	0.754	0.742
RSS	0.705	0.732	0.73	0.721	0.679
FSS	0.727	0.719	0.735	0.699	0.678
Baseline Results We offer three baseline methods, namely, UNIFORM, RSS (random single seller), and FSS (fixed single seller). UNIFORM purchases data points uniformly randomly from every sellers. RSS spends all budgets to buy as much as possible data points from one uniformly randomly chosen seller, while FSS does the same from a fixed seller. The baseline performance
can be found in Table 3. Overall, there is a large performance heterogeneity among the considered baselines. This underscores the necessity of carefully designed data acquisition strategies.
2.3.5 Adversarial Nibbler
The goal of the Adversarial Nibbler challenge is to engage the research community in jointly discovering a diverse set of insightful long-tail problems for text-to-image models and thus help identify current blindspots in harmful image production (i.e., unknown unknowns). We focus on prompt-image pairs that currently slip through the cracks of safety filters – either via intentful and subversive prompts that circumvent the text-based filters or through seemingly benign requests that nevertheless trigger unsafe outputs. By focusing on unsafe generations paired with seemingly safe prompts, our challenge zeros in on cases that (1) are most challenging to catch via text-prompt filtering and (2) have the potential to be harmful to non-adversarial end users.
Use-Case Rationale Building on recent successes for data fairness [23], quality [12], limitations [28, 58], and documentation and replication [42] of adversarial and data-centric challenges for classification models, we identify a new challenge for discovering failure modes in generative text-to-image models. Models such as DALL-E 2, Stable Diffusion, and Midjourney have reached large audiences in the past year owing to their impressive and flexible capabilities. While most models have text-based filters in place to catch explicitly harmful generation requests, these filters are inadequate to protect against the full landscape of possible harms. For instance, [45] recently revealed that Stable Diffusion's obfuscated safety filter only catches sexually explicit content but fails to address violence, gore, and other problematic content. Our objective is to identify and mitigate safety concerns in a structured and systematic manner, covering both the discovery of new failure modes and the confirmation of existing ones. Adversarial Nibbler exercises DataPerf's ability to host challenges focused on evaluating generative AI and AI safety, and demonstrates DataPerf's support for high-demand GPU inference tasks and integration with external APIs. Additionally, this challenge demonstrates new benchmark criterion targeted at generative models.
Benchmark Definition This competition is aimed at researchers, developers, and practitioners in the field of fairness and development of text-to-image generative AI. We intentionally design the competition to be simple enough that researchers from non-AI/ML communities can participate, though the incentive structure is aimed at researchers. Participants must write a benign or subversive prompt which is expected to correspond to an unsafe image. Our evaluation server returns several generated images using DataPerf-managed API licenses, and the participant selects an image (or none) that falls into one of our failure mode categories surrounding stereotypes, culturally inappropriate, or ethically inappropriate generations, among others.
We aim to collect prompts that are considered as a "backdoor" for unsafe generation. We focus on two different types of prompt-generation pairs, each reflecting a different user-model interaction mode. (1) Benign prompts with unexpected unsafe outputs. A benign prompt in most cases is expected to generate safe images. However, in some instances even a benign prompt may unexpectedly trigger unsafe or harmful generations. (2) Subversive prompts with expected unsafe outputs. While text filters catch unambiguously harmful requests, users can adversely bypass the filters via subversive prompts which trigger the model to produce unsafe or harmful generations. The data gathered from the first round is then sent to humans for validation before results are released to a leaderboard. Participants are rewarded based on two criteria: validated attack success – the number of unsafe images generated, and submission creativity – assessing coverage in terms of attack mode across lexical, semantic, syntactic, and pragmatic dimensions.
Baseline Results As the Adversarial Nibbler challenge focuses on crowdsourced data and deviates from the other benchmarks, there is no starter code or a baseline result. Instead, the goal is to analyze the data from the challenge submissions and create a publicly available dataset consisting of prompt-image pairs. These pairs that will undergo validation will be used to establish data ratings and will serve as a valuable resource for drawing conclusions and insights from the submissions received. Adversarial Nibbler has already collected several hundred unique prompts. Results from this challenge, consisting of a public dataset and insights to red teaming approaches from challenge participants, will be disseminated at the IJCNLP-AACL 2023 ART of Safety Workshop<sup>3</sup>.
<sup>3</sup>https://sites.google.com/view/art-of-safety/home
To ensure academic innovations have real-world impact, systems research in the machine learning industry has relied on benchmarking, including MLPerf [33, 46], DawnBench [10] and related efforts [19, 60, 51]. Data-centric benchmarking has similarly received increased focus. Zha et al. [59] surveys recent efforts, including benchmarks in AutoML [61], semi-supervised strategies [56], data selection [16], and data cleaning approaches [30]. Benchmark competitions have also emerged as a valuable comparative method in data-centric AI. DataComp [18] is a recent competition focused on filtering multimodal training data for language-image pairs, with a focus on improving accuracies under different fixed compute budgets. The Crowdsourcing Adverse Test Sets for Machine Learning (CATS4ML) Data Challenge [3] asked participants to find examples that are confusing or otherwise problematic for image classification algorithms to process, in which participants submitted misclassified samples from the Google Open Images dataset, identifying 15,000 adversarial examples. Drawing inspiration from these efforts, DataPerf solicits user-contributed benchmarks by providing an extensible platform for hosted public challenges and leaderboards, with long-term, industry-guided support for benchmarks through the DataPerf Working Group and MLCommons.
Several existing benchmarks evaluate state-of-the-art methods in selection. For instance, prior work in benchmarking high-dimensional feature selection [8] and augmentation strategies [38] are conceptually similar to the vision selection and roman numeral tasks. DCBench [16] is a benchmark and Python API for fixed-budget cleaning, slice discovery [17], and coreset selection[11], which are applicable to our speech selection, vision selection, and data debugging tasks. The baselines in DataPerf do not exhaustively compare all state-of-the-art data-centric methods, but instead encourage students and new practitioners to apply existing methods from the literature, while still enabling academic researchers to propose novel methods. Persistent online leaderboards for each challenge enable new solutions to be compared to all prior submissions. The DataPerf Working Group endeavors to solicit new challenges from the data-centric research community, and to integrate existing benchmarks (ideally in partnership with their respective authors) in additional domains, such as active learning for tabular data [36], label uncertainty [41], and noisy annotations [49].
4 Statement of Ethics
Dynabench collects self-declared usernames and email addresses during registration, and these usernames may correspond to personal identifiable information. Dynabench also collects uploaded artifacts during submission which can optionally be viewed by other users as open benchmark results.
Adversarial Nibbler requires additional guidelines for participants as it collects potentially sensitive content of harmful and disturbing depictions which may negatively impact participants and raters. These guidelines follow best practices for protecting well-being [27] and provides communication with challenge organizers, preparation for working with potentially unsafe imagery, and external resources for psychological support (detailed in Appendix A.7)
The purpose of DataPerf is to improve machine learning by expanding AI research from just models to models and datasets. The benchmarks aim to improve standard practices for dataset development, and add rigor to assessing the quality of training and test sets, across a wide variety of ML applications. Systematic dataset benchmarking is vital, per the adage "what gets measured gets improved." The initial version of DataPerf comprises five benchmarks, each with unique rules, evaluation methods, and baseline implementations, and an open-source, extensible evaluation platform.
DataPerf will continue to expand by adding additional benchmarks to the suite, with input and contributions from the community. Additionally, in order to increase the reproducibility of challenges and expand the scope of the evaluation, we plan to add a 'Closed Division' where participants must submit an algorithm that is then evaluated on a 'hidden training set', meaning it is tested on data that the submitter has never seen. This evaluates if the algorithm can generalize beyond the original dataset's distribution. We urge interested parties to join the DataPerf Working Group, and to participate in and contribute to current benchmarking challenges or propose new challenges at https://dataperf.org.
[51] F. Tang, W. Gao, J. Zhan, C. Lan, X. Wen, L. Wang, C. Luo, Z. Cao, X. Xiong, Z. Jiang, et al. Aibench training: Balanced industry-standard ai training benchmarking. In 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 24-35. IEEE, 2021.
[52] TAUS. Taus data marketplace, BloombergAPI. (Accessed on 05/22/2023).
[53] M. Tsuchiya. Performance impact caused by hidden bias of training data for recognizing textual entailment. Proceedings of the Eleventh International Conference on Language Resources and Evaluation, 2018.
[54] Twitter. Twitter api, 2023. (Accessed on 05/22/2023).
[55] E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh. Universal adversarial triggers for attacking and analyzing nlp. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
[56] Y. Wang, H. Chen, Y. Fan, W. Sun, R. Tao, W. Hou, R. Wang, L. Yang, Z. Zhou, L.-Z. Guo, et al. Usb: A unified semi-supervised learning benchmark for classification. Advances in Neural Information Processing Systems, 35:3938-3961, 2022.
[57] D. Weissenborn, G. Wiese, and L. Seiffe. Making neural QA as simple as possible but not simpler. In R. Levy and L. Specia, editors, Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017, pages 271-280. Association for Computational Linguistics, 2017.
[58] C. Welty, P. Paritosh, and L. Aroyo. Metrology for ai: From benchmarks to instruments. arXiv preprint arXiv:1911.01875, 2019.
[59] D. Zha, Z. P. Bhat, K.-H. Lai, F. Yang, Z. Jiang, S. Zhong, and X. Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158, 2023.
[60] H. Zhu, M. Akrout, B. Zheng, A. Pelegris, A. Phanishayee, B. Schroeder, and G. Pekhimenko. TBD: Benchmarking and analyzing deep neural network training. arXiv preprint arXiv:1803.06905, 2018.
[61] M.-A. Zöller and M. F. Huber. Benchmark and survey of automated machine learning frameworks. Journal of artificial intelligence research, 70:409-472, 2021.
A.1 Terminology for Training Sample Selection
In this section, for convenience, we clarify the terminology related to training sample selection used in our challenges, where (in accordance with widely-used terminology) a training sample is an individual data point in a dataset. Sec. 2.3 clarifies our distinction between challenges, benchmarks, and leaderboards.
- Training set selection: this task refers to choosing a small set of samples for training a model from a larger pool of potentially noisy training data. This task is also commonly referred to as coreset selection.
- Training IDs are integer enumerations of training data samples ([1,2,3,...]), or unique strings each corresponding to a file containing data for an individual sample ([audiol.wav, audio2.wav, ...])
- Allowed training IDs: This term refers to the list of potential samples which can be included in a proposed coreset by a challenge participant. In other words, this is the full list of training IDs, which participants can form subsets of.
- Selected training IDs: this is a concretized coreset, submitted to the DataPerf online platform for evaluation. In other words, selected training IDs are a subset of training IDs drawn from the full list of allowed training IDs. This is indicated as "New Train Set" in Figure 1.
A.2 Reproducibility
Source code for the inaugural DataPerf challenges is hosted at github.com/mlcommons/dataperf. We use git submodules to reference a fixed commit hash of the respective parent repositories for each challenge. This preserves flexibility for a diverse set of challenges and allows challenge owners to maintain control of their benchmarks and promote community visibility within their own GitHub organizations while simultaneously ensuring the challenges remain static during the competition and are archived as-is with respect to each round of challenges.
We additionally provide links to each benchmark's repository here, containing code and documentation for reproducibility.
1. Selection for Speech: The baseline for the speech training set selection benchmark is available at https://github.com/harvard-edge/dataperf-speech-example
2. Selection for Vision: The baseline for the vision training set selection benchmark will be available at https://github.com/CoactiveAI/dataperf-vision-selection, we are in the process of releasing the code.
3. Debugging for Vision: The vision debugging baseline is available at https://github.com/DS3Lab/dataperf-venture-debugging
4. Data Acquisition: The data acquisition baseline is available at https://github.com/facebookresearch/Data_Acquisition_for_ML_Benchmark
5. Adversarial Nibbler: As the Adversarial Nibbler challenge focuses on crowdsourced data there is no starter code or a baseline results for participants. The server code for the challenge is available as part of Dynabench (Sec. 2.2) at https://github.com/mlcommons/dynabench
In the following sections, to provide a fixed reference, we include extended documentation for each challenge reproduced from each of their respective source-code repositories, as of August 2023, which reflects the challenge requirements and evaluation structure for all inaugural challenges in the DataPerf suite. Though future training set selection and debugging challenges in DataPerf may diverge from some of the technical specifications provided here, we emphasize that these challenges as described can also serve as fixed benchmarks by the data-centric AI community, and future solutions can be submitted to the leaderboards for these rounds of challenges in adherence to these specifications and rules.
A.3 Selection for Speech