SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Fig. 4, we provide the number of training and evaluation sample counts available for each target keyword, and the nontarget data, for the three languages in the benchmark. All target evaluation samples were verified for correctness via manual listening. For each language, a participant trains a six category (five target words and one nontarget category) model, using a maximum of 25 or 60 samples drawn from the training pool. Evaluation proceeds by training ten models using ten random seeds, and for each model, reporting the macro F1 score on all evaluation samples for target and nontarget words for each language.
English
Target Keywords	Training Samples	Eval Samples
episode	565	85
job	1261	239
fifty	819	163
route	640	124
restaurant	647	122
Total samples	3932	733
Portuguese
Target Keyword	Training Samples	Eval Samples
pessoas (people)	1042	251
grupo (group)	383	95
camisa (shirt)	354	93
tempo (time)	375	95
andando (walking)	320	79
Total samples	2474	613
Indonesian
Target Keyword	Training Samples	Eval Samples
karena (because)	181	25
sangat (very)	159	42
bahasa (language)	135	37
belajar (study)	107	28
kemarin (yesterday)	103	45
Total samples	685	177
Nontarget data	Training Samples	Eval Samples
Number of words	100	300
Samples per word	100	100
Total samples	10000	30000
Nontarget data	Training Samples	Eval Samples
Number of words	100	300
Samples per word	50	50
Total samples	5000	15000
Nontarget data	Training Samples	Eval Samples
Number of words	100	300
Samples per word	15	15
Total samples	1500	4500
Figure 4: Target keywords and sample counts for speech selection.
We reproduce documentation from https://github.com/harvard-edge/dataperf-speech-example/ as a centralized reference here.
Dataperf-Selection-Speech<sup>4</sup> is a challenge hosted by DataPerf.org<sup>5</sup> that measures the performance of dataset selection algorithms. The model training component is frozen and participants can only improve the accuracy by selecting the best training set. The benchmark is intended to encompass the tasks of dataset cleaning and coreset selection for a keyword spotting application. As a participant, you will submit your proposed list of training samples to the leaderboard on DynaBench<sup>6</sup> where the model is trained, evaluated, and scored.
Evaluation Metric
You are given a training dataset for spoken word classification and your goal is to produce an algorithm that selects a subset of size M examples (a coreset) from this dataset. Evaluation proceeds by subsequently training a fixed model (sklearnensemble.VotingClassifier with various constituent classifiers) on your chosen subset, and then scoring the model's predictions on fixed test data via the sklearn.metrics.f1_score metric with average = macro. We average the score over 10 random seeds (located in workspace/datapaperf_speech_config.yaml) to produce the final score. M is user defined, but Dynabench will host two leaderboards per language with coreset size caps of 25 and 60.
For each language, the challenge includes two leaderboards on Dynabench (six leaderboards in total). Each leaderboard corresponds to a language and a fixed maximum number of training samples (your submission can specify fewer samples than the maximum coreset size).
The training dataset consists of embedding vectors produced by a pretrained keyword spotting model $^{7}$  (model checkpoint weights $^{8}$ ) for five target words in each of three languages (English, Portuguese, and Indonesian) taken from the Multilingual Spoken Words Corpus $^{9}$ . The classifier also
<https://www.dataperf.org/training-set-selection-speech> <https://dataperf.org> <https://dynabench.org/tasks/speech-selection> <https://arxiv.org/abs/2104.01454> <https://github.com/harvard-edge/multilingual_kws/releases/download/v0.1-alpha/multilingual_context_73_0.8011.tar.gz> <https://mlcommons.org/words>
includes a nontarget category representing unknown words which are distinct from one of the five target words. To train and evaluate the classifier's ability to recognize nontarget words, we include a large set of embedding vectors drawn from each respective language.
Solutions should be algorithmic in nature (i.e., they should not involve human-in-the-loop audio sample listening and selection). We warmly encourage open-source submissions. If a participant team does not wish to open-source their solution, we ask that they allow the DataPerf organization to independently verify their solution and approach to ensure it is within the challenge rules.
Our introductory notebook on Google Colab is available at
https://colab.research.google.com/gibth/harvard-edge/dataperf-speech-example/blob/main/dataperf_speech_colab.ipynb10
This colab walks through performing coreset selection with our baseline algorithm<sup>11</sup> and running our evaluation script<sup>12</sup> on the coresets for English, Portuguese, and Indonesian.
Below, we provide additional documentation for each step of the above colab (downloading, training coreset selection, and evaluation).
Please see the challenge rules on datapferf.org<sup>13</sup> for more details - in particular, we ask you not to optimize your result using any of the challenge evaluation data. Optimization (e.g., cross-validation) should be performed on the samples in allowed_training_set.yaml for each language, and solutions should not be optimized against any of the samples listed in eval.yaml for any of the languages.
Since this speech challenge is fully open, there is no hidden test set. A locally-computed evaluation score is unofficial, but should match the results on DynaBench, and included here solely to allow for double-checking of DynaBench-computed results only if necessary. Official evaluations will only be performed on DynaBench. The following command performs local (offline) evaluation:
python eval.py --language en --train_size 25
This will output the macro f1 score of a model trained on the selected training set, against the official evaluation samples.
To develop their own selection algorithm, participants should: - Create a new selection.py algorithm in selection/implementations which subclasses TrainingSetSelection $^{14}$  - Implement select() in your class to use your selection algorithm - Change selection_algorithm_module and selectionALAgorithm_class in workspace/dataperfSpeech_config.yaml to match the name of your selection implementation - optionally, add experiment config to workspace/dataperf_speech_config.yaml (this can be accessed via self.config in ) - Run your selection strategy and submit your results to DynaBench
Once participants are satisfied with their selection algorithm they should submit their {lang} _ {size} _ {train}.json files to DynaBench<sup>15</sup>. A seperate file is required for each language and training set size combination (6 total).
Each supported language has the following files:
$^{10}$  <https://colab.research.google.com/github/harvard-edge/datapaperf-speech-example/blob/main/datapaperf_speech_colab.ipynb>  
 $^{11}$  <https://github.com/harvard-edge/datapaperf-speech-example/blob/main/selection/implementations/baseline_selection.py>  
 $^{12}$  <https://github.com/harvard-edge/datapaperf-speech-example/blob/main/eval.py>  
 $^{13}$  <https://dataperf.org>  
 $^{14}$  <https://github.com/harvard-edge/datapaperf-speech-example/blob/main/selection/selection.py#L16>  
 $^{15}$  <https://dynabench.org/tasks/speech-selection>
- train_vectors : The directory that contains the embedding vectors that can be selected for training. The file structure follows the pattern train_vectors/en/left.parquet. Each parquet file contains a clip_id column and a mswc embedding vector column.
- eval_vectors : The directory that contains the embedding vectors that are used for evaluation. The structure is identical to train vectors
- allowed_train_set.yaml : A file that specifies which sample IDs are valid training samples. The file constrains the following structure {"targets": {"left": [list]}, "nontargets": [list]}
- eval.yaml : The evaluation set for eval.py. It follows the same structure as allowed_train_set.yaml. Participants should never use this data for training set selection algorithm development.
- {lang}__{size}__train.json: The file produced by selection:main that specifies the language specific training set for eval.py.
All languages share the following files: * datapaperf_speech_config.yaml : This file contains the configuration for the datapaperf-speech-example workflow. Participants can extend this configuration file as needed.
- We ask you to please not look at or use the provided evaluation sets in any way other than for offline evaluation of your submissions to Dynabench (e.g., do not optimize on the evaluation data).
- Each training set in the final submission will be capped at either 25 or 60 samples, depending on the leaderboard. Training sets with more than the maximum number of selected samples for that leaderboard will be rejected.
- For this challenge, the submitted train.json file can be unbalanced, therefore an optimal solution may leverage an unbalanced training set.
- The provided candidate pool of training samples is a custom subset of the Multilingual Spoken Words Corpus (MSWC). You may analyze other languages in MSWC, but please do not use English, Portuguese, or Indonesian MSWC data outside of the samples specified in allowed training set. yaml for each respective language.
A.4 Selection for Vision
We reproduce documentation from https://github.com/CoactiveAI/dataperf-vision-selection as a centralized reference here.
Our github repo serves as the starting point for offline evaluation of submissions for the training data selection visual benchmark. The offline evaluation can be run on both your local environment as well as a containerized image for reproducibility of score results.
For a detailed summary of the a benchmark, refer to the provided documentation<sup>16</sup>.
Creating a submission
A valid submission for the open division includes the following:
- A description of the data selection algorithm/strategy used
- A training set for each classification task as specified below
- (Optional) A script of the algorithm/strategy used
Each training set file must be a .csv file containing two columns: ImageID (the unique identifier for the image) and Confidence (the binary label, either a 0 or 1). The ImageIDs in the training set files must be limited to the provided candidate pool of training images (i.e. ImageIDs in the downloaded embeddings file).
The included training set file serves as a template of a single training set:
<https://www.dataperf.org/training-set-selection-vision>
cat dataperf-vision-selection/data/train_sets/random_500.csv
ImageID, Confidence  
0002643773a76876, 0  
0016a0f096337445, 0  
0036043ce525479b, 1  
00526f123f84db2f, 1  
0080db2599d54447, 1  
00978577e9fdd967, 1
Offline evaluation
The configuration for the offline evaluation is specified in task_setup.yaml file. For simplicity, the repo comes pre-configured such that for offline evaluation you can simply:
1. Copy your training sets to the template filesystem
2. Modify the config file to specify the training set for each task
3. Run offline evaluation
4. See results in stdout and results file in data/results/
1. Copy training sets for each task  
cd datapaperf-vision-selection  
cp /path/to/your/training/sets/Cupcake.csv data/train_sets/  
cp /path/to/your/training/sets/Hawk.csv data/train_sets/  
cp /path/to/your/training/sets/Sushi.csv data/train_sets/
2. task_setup.yaml: modify the training set relative path for each classification task Cupcake: ['train_sets/Cupcake.csv', 'test_sets/alpha_test_set_Cupcake_256.parquet'] Hawk: ['train_sets/Hawk.csv', 'test_sets/alpha_test_set_Hawk_256.parquet'] Sushi: ['train_sets/Sushi.csv', 'test_sets/alpha_test_set_Sushi_256.parquet']
3a. Run offline evaluation (docker) docker-compose up --build --force-recreate
3b. Run offline evaluation (local python)  
python3 main.py
4. See results (file will have save timestamp in name)  
cat data/results/resultUTC-2022-03-31-20-19-24.json
{ "Cupcake": { "accuracy": 0.5401459854014599, "recall": 0.463768115942029, "precision": 0.5517241379310345, "f1": 0.5039370078740157 }, "Hawk": { "accuracy": 0.296551724137931, "recall": 0.16831683168316833, "precision": 0.4857142857142857, "f1": 0.2500000000000006 }, "Sushi": {
"accuracy": 0.5185185185185185, "recall": 0.6261682242990654, "precision": 0.638095238095238, "f1": 0.6320754716981132 } }
Evaluation Criteria In this challenge, your task will be to design a data selection strategy that chooses the best training examples from a candidate pool of training images (a custom subset of the Open Images Dataset V6 training set) which maximizes the F1 score across a set of binary classification tasks for different visual concepts (e.g., "Cupcake", "Hawk", "Sushi"). Your submission will be a training set for each of the classification tasks in this challenge.
1. We ask you to please not look at or use the provided test sets in any way other than for offline evaluation.
2. We ask you to only use the provided data for developing your solution (unless otherwise explicitly stated).
3. Submissions that rely on human intervention are allowed. The intervention strategy must be clearly explained such that the results are as reproducible and extensible as possible.
4. Algorithmic submissions may not rely on external intervention (e.g. humans, extra data). The results should be reproducible and extensible to other datasets.
5. Your developed solution should be practical and reasonably efficient given the scope of the challenge (e.g., your algorithm shouldn't perform an exhaustive search).
6. Rules regarding participation:
- Participants can only belong and participate in one team
- Individuals are considered a team
- Teams must be defined before the end of the challenge
Each team must have a leader who is responsible for submissions to the online evaluation platform
- Participants should not access or inspect submissions or selection code from other participating teams until after the challenge concludes
7. Each training set that is part of the final submission will be limited to 1,000 data points. Training sets with more than 1,000 ( imageID, label) pairs will be rejected
8. The provided candidate pool has no labels, and as such, part of the challenge involves using the information contained in the embeddings as effectively as possible.
9. The provided candidate pool is a custom subset of the training set for the Open Images dataset. You may refer to metadata from the Open Images dataset.
10. If needed, you can leverage the human-verified and/or machine generated labels available in the metadata from the Open Images dataset. However, we encourage creative solutions that minimize the amount of labels used.
A.5 Debugging for Vision
We reproduce documentation from https://github.com/DS3Lab/dataperf- vision-debugging as a centralized reference here.
Training Set Cleaning
When dealing with massive datasets, noises in the datasets become inevitable. This is increasingly the problem for ML training and noises in the dataset can come from many places:
- Natural noises come in during data acquisition.
- Algorithmic labeling: e.g., weak supervision, and automatically generated labels by machines.
Data collection biases (e.g., biased hiring decisions).
If trained over these noisy datasets, ML models might suffer not only from lower quality, but also potential risks on other quality dimensions such as fairness. Careful data cleaning can often accommodate this, however, it can be a very expensive process if we need to investigate and clean
all examples. By using a more data-centric approach we hope to direct human attention and the cleaning efforts toward data examples that matter more to the improvement of ML models.
In this data cleaning challenge, we invite participants to design and experiment data-centric approaches towards strategic data cleaning for training sets of an image classification model. As a participant, you will be asked to rank the samples in the entire training set, and then we will clean them one by one and evaluate the performance of the model after each fix. The earlier it reached a high enough accuracy, the better your rank is.
DataPerf currently hosts an open division challenge for the vision debugging challenge. In the open division, you will submit the output of running your cleaning algorithm on a given dataset. Then we will train the model and evaluate it based on your submission. As future work, we will include a closed division, where you will submit the cleaning algorithm itself, and we will run your algorithm to generate the output on several hidden datasets. Then we evaluate your submissions.
How to Participate
In order to make participation as easy as possible, we've come up with a set of tools that ease the process of iterating and submitting: MLCube<sup>17</sup> and Dynabench<sup>18</sup>. MLCube was developed to help you get started on your local computer, and it will help you download the datasets, run some baseline algorithms, evaluate your submission and baselines and plot the results. Once you are satisfied with your results, you can then submit it to Dynabench, which is a platform where we will evaluate your submission and show the leaderboard for this challenge.
Offline Evaluation with MLCube
The evaluation code of the challenge is entirely open at https://github.com/DS3Lab/dataperf-vision-debugging, where you can run some baselines and evaluate your algorithms locally. Below are the instructions on how to setup the environment and run them locally.
In order to evaluate your own algorithms, you can either:
- Provide a .txt file, as described in https://github.com/DS3Lab/dataperf- vision-debugging#open-division-creating-a-submission19. Place it under the workspace/submissions folder. It will be evaluate by the evaluate command.
- Write an algorithmic approach in the app/baselines-debugging.py. It will be run and evaluate together with other baseline approaches.
Online Evaluation with Dynabench
As stated before, for the open division we ask that you submit multiple files, each being the output of the cleaning algorithm you developed. The only limitations on your submission is:
- each training file should have exactly 300 examples, which is the size of the training set.
- and that you must submit to all evaluating classes at the same time.
Evaluation Metric Your submission will be evaluated based on "how many samples your submission needs to fix, to achieve a high enough accuracy". This is to imitate real use cases of the data cleaning algorithms, where we want to inspect as less samples as possible, but keep the data quality good enough. For example, if the accuracy of the model, trained on a perfectly clean dataset, is 0.9, then we define the high enough accuracy to be  $0.9 * 95\% = 0.855$ . Assume that algorithm A achieves an accuracy of 0.855 after fixing 100 samples and algorithm B achieves an accuracy of 0.855 after fixing 200 samples, then score(A) = 100/300 = 1/3 while score(B) = 2/3. In other words, the lower the score, the better the cleaning algorithm.
1. We ask you to please not look at or use the provided test sets in any way other than for offline evaluation.
<https://mlcommons.org/en/mlcube/> <https://mlcommons.org/en/groups/research-dynambench/> <https://github.com/DS3Lab/dataperf-vision-debugging# open-division-creating-a-submission>
2. We ask you to only use the provided data for developing your solution (unless otherwise explicitly stated).
3. Algorithmic submissions may not rely on external intervention (e.g. humans, extra data). The results should be reproducible and extensible to other datasets.
4. Your developed solution should be practical and reasonably efficient given the scope of the challenge (e.g., your algorithm shouldn't perform an exhaustive search).
5. Rules regarding participation:
- Participants can only belong and participate in one team
- Individuals are considered a team
- Teams must be defined before the end of the challenge
Each team must have a leader who is responsible for submissions to the online evaluation platform
- Participants should not access or inspect submissions or selection code from other participating teams until after the challenge concludes
6. Each training set that is part of the final submission will be limited to 1,000 data points. Training sets with more than 1,000 ( imageID, label) pairs will be rejected
7. For this challenge, the provided candidate pool (i.e. embeddings) has no labels, and as such, part of the challenge involves using the information contained in the embeddings as effectively as possible.
8. The provided candidate pool is a custom subset of the training set for the Open Images dataset. You may refer to non-labels metadata from the Open Images dataset (https://storage.googleapis.com/openimages/web/download.html)
A.6 Data Acquisition
We reproduce documentation from https://github.com/facebookresearch/Data_Acquisition_for_ML_Benchmark as a centralized reference here.
The github repo serves as the starting point for submissions and evaluations for data acquisition for machine learning benchmark, or in short, DAM, as part of the DataPerf benchmark suite https://dataperf.org/20
dataperf-dam: A Data-centric Benchmark on Data Acquisition for Machine Learning
1. What is the DAM benchmark?
An increasingly large amount of data is purchased for AI-enabled data science applications. How to select the right set of datasets for AI tasks of interest is an important decision that has, however, received limited attention. A naive approach is to acquire all available datasets and then select which ones to use empirically. This requires expensive human supervision and incurs prohibitively high costs, posing unique challenges to budget-limited users.
How can one decide which datasets to acquire before actually purchasing the data to optimize the performance quality of an ML model? In the DAM (Data-Acquisition-for-Machine-learning) benchmark, the participants are asked to tackle the aforementioned problem. Participants need to provide a data purchase strategy for a data buyer in K (=5 in the beta version) separate data marketplaces. In each data marketplace, there are a few data sellers offering datasets for sale, and one data buyer interested in acquiring some of those datasets to train an ML model. The seller provides a pricing function that depends on the number of purchased samples. The buyer first decides how many data points to purchase from each seller given a data acquisition budget b. Then those data points are compiled into one dataset to train an ML model f(). The buyer also has a dataset Db to evaluate the performance of the trained model. Similar to real-world data marketplaces, the buyer can observe no sellers' datasets but some summary information from the sellers.
2. How to participate this challenge?
20<https://dataprf.org/>
We suggest to start participating by using the colab notebook $^{21}$ . It is self-contained, and shows how to (i) install the needed library, (ii) access the buyer's observation, and (iii) create strategies ready to be submitted. In the following we explain this in more details.
3. How to access the buyer's observation?
We provide a simple python library to access the buyer's observation in each data marketplace. For example, to specify the marketplace id, one can use
from dam import Dam MyDam  $=$  Dam.instance  $\equiv 0$
The following code lists the buyer's budget, dataset, and ml model.
budget  $=$  MyDam.getbudget()   
buyer_data  $\equiv$  MyDam.getbuyerdata()   
mlmodel  $=$  MyDam.getmlmodel()
To list all sellers' ids, execute
sellers_id = MyDam.getsellerid()
To get seller i's information, run
seller_i_price, seller_i_summery, seller_i_samples  $=$  MyDam.getsellerinfo(seller_id=i)
sellerprice contains the pricing function. sellerissummary includes (i) the number of rows, (ii) the number of columns, (iii) the histogram of each dimension, and (iv) the correlation between each column and the label. Sellerisamples contains 5 samples from each dataset.
Note: For simplification purposes, all sellers sell the same type of data, or in a more mathematically way, their data distribution shares the same support. For example, the number of columns are the same, and so the semantic meaning.