Data Checklist: On Unit-Testing Datasets with Usable Information
Authors: Heidi Chenyu Zhang
Shabnam Behzad
Kawin Ethayarajh
Dan Jurafsky
Venue: COLM (2024)
Keywords: data checklist, usable information, dataset artifact, preference alignment
TL;DR: We propose a data checklist consisting of 10 unit tests that diagnose dataset artifacts and ground model behavior in the data they are trained on.

Abstract: Model checklists (Ribeiro et al., 2020) have emerged as a useful tool for understanding the behavior of LLMs, analogous to unit-testing in software engineering. However, despite datasets being a key determinant of model behavior, evaluating datasets -- e.g., for the existence of annotation artifacts -- is largely done ad hoc, once a problem in model behavior has already been found downstream.
In this work, we take a more principled approach to unit-testing datasets by proposing a taxonomy based on the $\mathcal{V}$-information literature. We call a collection of such unit tests a data checklist.
Using the checklist, not only are we able to recover known artifacts in well-known datasets such as SNLI, but we also discover previously unknown artifacts in preference datasets for LLM alignment.
Data checklists further enable a new kind of data filtering, which we use to improve the efficacy and data efficiency of preference alignment.

Published as a conference paper at COLM 2024
Data Checklist: On Unit-Testing Datasets with Usable Infor- mation
Heidi C. Zhang Stanford University chenyuz@stanford.c edu
Shabnam Behzad Georgetown University srabnam@cs.georgetoom edu
Kawin Ethayarajh Stanford University kawinêstanford edu
Dan Jurafsky Stanford University jurafsky@stanford.ced
Abstract
Model checklists (Ribeiro al. 2020) have emerged as useful tool for understanding the behavior of LLMs, analogous to unit-testing in software engineering. However, despite datasets being key determinant of model behavior, evaluating datasets- e.g., for the existence of annotation artifacts is largely done ad hoc, once problem model behavior has already been found downstream. In this work, we take more principled approach to unit-testing datasets by proposing taxonomy based on the V-informattin literature. We call collection of such unit tests data checklist. Using checklist, not only are we able to recover known artifacts in well-known datasets such SNLI, but also discover previously unknown artifacts in preference datasets for LLM alignment. Data checklists further enable a new kind of data filtering, which we to improve the efficacy and data efficiency of preference alignment.
Introduction
Behavioral checklists have been proposed to test the capabilities and limitations of NLP models beyond traditional benchmarks (Ribeiro al., 2020), and they have been widely evaluation (Bhatt et al. 2021).
used in red teaming (Perez et al. (202) and human-in-the- However, model checklists do not address the fact that many bugs observed during evalus tion can be ascribed to the dataset that the model is trained or fine-tuned on, suggesting that these problems could be mitigated by audits further upstream.
We turn to the notion of V-information the primitive for such audits. Written as Iv(X Y), this measures how much information an input variable contains about the output that is accessible to model family V (Xu et al., 2019). Recently, it has emerged tool for understanding why datasets are often more simplistic than the real-world tasks they purport to reflect, with dataset difficulty being framed the lack of usable information (Ethayarajh et al.) 2022). This can be used to compare different function families, datasets, data points drawn from the same distribution, and even different features of the input.
popular
In this work, we build on these primitives by introducing a taxonomy of unit tests for data, which explicitly maps common classes of questions about datasets into structured expressions of usable information. One common type of question is whether particular feature the only part of the input that useful. For example, given toxicity detection dataset, we would want to know whether profane words in the input are sufficient to predict toxicity; that would be undesirable for a toxicity benchmark, as simple n-gram model could solve the task. This question can be articulated more formally as: "Is there any V-information that X contains about Y that not already contained in feature ((X)?, or as
Code for the checklists available at ahttpp///ghhhaoooo////eeeeehhhia checklist

---

Published as a conference paper at COLM 2024
Conditioned Out
Predictor for Y
Something can be learned without +
X
Egg Profane words (4) insuffieient
HHH HHH-hormless.
Insufficiency
Something be learned without a²?
0
Necessity
E.g. Response length (4b is
None
Applicability
Non-Exclusivity
Viability
pecessory in SHP.
Can something be learned?
Something can learned with Something can be leurned with 4"? R.g. Premise-hyppp hypothesiss overlop (0) is E.g. English words erchesively useful in SNLL.
F.g. SNLJ is vighle, containing 0.97 bits of V-info.
useful in Ultrafeedback.
Figure 1: Overview of the taxonomy of tests, where is an attribute to test for and '' is its complement. Each test is the form [v(A - YBB), where A the predictor variable, is the conditioned out variable, and ly denotes an estimate of the usable information. The viability, applicability, necessity, non-exclusivity, and insufficiency tests pass when the estimate is greater than ("something can be learned ..."), whereas their logical negations pass when the estimate less than ("nothing (or not enough) can be learned ...").
the inequality 1x(X - Y| where & contains all the profanity in the input X. In our taxonomy, this is called an insufficiency test, since & must be insufficient for predicting Y.
We formulate a taxonomy of 10 tests to help understand datasets from different angles. Checklists assembled from these tests provide systematic way to reason about dataset artifacts found in prior literature and allow us to discover artifacts in the more challenging and less well-studied case of preference alignment datasets. We further propose using the checklist inform data filtering, where we (1) filter out undesirable data instances to mitigate artifacts, and (2) improve efficiency by using less training data, while preserving test performance.
Our main contributions are as follows:
We propose taxonomy that maps common types of questions about data to one corresponding unit tests. Each unit test asserts that a particular (in)equality holds, where the LHS is a structured expression of usable information and the RHS is a tolerance level 0. We call a collection of these unit tests data checklist.
We provide a library for easy creation and application of data checklists across various models and datasets. Unlike prior libraries, which only support the es- timation of V-information when the output is a single categorical variable (e.g., sentiment), we support the setting when the output is sequence, as in open-ended text generation. Given the contemporary framing of all tasks as text-to-text (Raffel et al. 2020), our library has much more practical utility.
Using the checklist, we not only recover known annotation artifacts in the literature GGurraagaa et al.) 2018), but also discover previously unknown artifacts. For example, we find that the difference in length between text responses common artifact in preference datasets, which could explain why closed-source models such as GPT-4 prefer longer texts when used in an LLM-as-a-judge setup (Zheng et al., 2023). We then show that filtering examples based on their pointwise V-informtionn betore alignment leads to improvements in the performance of the aligned LLM.
2 Background
Y-information. Written ly(X - Y), the V-usable information (or V-information) is measure of how much information X includes about when constrained to predictive family V (Xu et al., 2019 Hewitt et al. 2021 |Ethayarahh et al. 2022). Most neural models
2

---

Published as a conference paper at COLM 2024
fine-tuned without any frozen parameters fulfill the definition of predictive family When V is the set of all functions, this is equivalent to Shannon mutual information. However, because are usually working with model families that are boundedly large, the amount of information that is usable is less than the Shannon mutual information.
We can apply any feature function & on the input X. In many cases [y( - Y) > ly(X - Y); this does not violate the data processing inequality because the feature does not create any new information; it can only make existing information more usable, much in the way that it is easier to learn from embeddings than it is from n-grams.
As originally defined in Xu et al. (2199) and adapted by Ethayarajh et al. (2022), V- information is computed as:
(x Y)
(1)
where X and Y are random variables with sample spaces X' and V, V is predictive family (V& = (f : u Ø - P()))), and the V-entropy of defined as:
Hy(Y)
inf feV
[[-log, [[]]()]
(2)
in which 2 is a null input that includes no information about Y. We measure the entropies in bits of information (as opposed to nats) which why log, used.
The conditional V-entropy is defined as:
Xy(Y inf log, [[X] feV
The higher the V-information, the easier the dataset can be said to be for the model family V [Ethayarajh et al. 2022).
In practice, if the model families are neural networks, one cannot necessarily find the best f f V, and instead models are trained to minimize the (conditional) entropy of Y using the training data, after which the (conditional) entropy is estimated using some held out test data. Therefore, in practice, we are calculating the empirical V-information ly(X - Y), which is boundedly close to the true V-information (see Xu al. 2019) for PAC bounds).
Pointwise difficulty. V-information can give an estimate of the overall difficulty of dataset, but what we wanted to estimate how difficult a given data point is relative to the distribution it is drawn from? To measure this, Ethayarajh et al. (0022) proposed pointwise V-information (PV1), which they define for an instance yx as:
PVI(x - y) = - log, g( + logg 8 [x](y)
(4)
where g'g' are the models that minimize the (conditional) label entropy.
In most of our experiments, V is the T5-base family, and g',g the checkpoints after finetuning T5 with and without the input, respectively. PVI(x - y) is estimated as the difference in the log-probability of g' and g on the gold label for a held-out instance. Similar to V-information, instances with higher PVI are interpreted as being easier for the model family V.
Conditional Y-information. Lastly, we are often interested in measuring how much information X contains about Y in excess of what already in some feature c (e-g., its length). Hewitt et al. (2021) proposed conditional V-information, in which one can condition out different attributes. Conditional Y-information defined as:
(5) In practice, x(X) u {X] can be obtained by concatenating the input text (X) and the text
(x Y|)(X)) = Hy
Hv(Y) u (X))
resulting from applying the attribute to the input text (& ()) We also introduce the notion of a feature complement (+/ (t((X)), which is everything in X that is not in ((X).
2 should satisfy optional ignorance; meaning the side information (X) can be ignored by the agent if it chooses to. See Xu et al. 2201 Hewitt al. (2021) for more details.
3

---

Published as a conference paper at COLM 2024
Formula
Test Type
Description
1 -)
> E
Viability
X contains some usable information.
< e
Unviability
X contains no usable information.
> E
Applicability
should be possible to learn something using attribute O
[v(1(X) Y)
<
Inapplicability
It should not be possible to learn anything using attribute @
[v(0((x) Y)
> E
Non-Exclusivity Exclusivity
Not all usable information exists only in attribute ④.
All usable information exists only in attribute 4
Insufficiency
Not all usable information that X contains about Y contained in 4; there is some outside X.
1x(x - x(0(x))
< Sufficiency
All usable information that X contains about is
also contained in cb (though not necessarily exclusive to O)).
> Necessity
Some usable information exists only in attribute +b attribute & is necessary to extract all usable information from X).
- Yx(xx)
< Redundancy
No usable information exists only in attribute + (i.e., the usable information in O is redundant).
Table 1: Proposed taxonomy of data checklists. X is the input variable, Y is the output, Φ is feature and V is the model family. e is small, non-negative tolerance threshold. In practice, we use desirable if the task is solvable via detecting profanity alone, so an insufficiency test with @ proffniity (which masks out non-profane words) should pass.
0.01. Consider a hate speech detection task as an example: it is not
3 Data Checklists
Based on V-information, we propose taxonomy for various types of tests that could be used to understand the difficulty of given dataset and features of interest. To create these tests, X, Y, and & are the variables, but the model (V) stays the same. Our proposed tests address common questions about the target dataset. Each unit test an assertion of particular inequality: LHS or LHS < E, where LHS is an expression of usable information and e is a small non-negative tolerance threshold.
We call a collection of these unit tests data checklist. There are ten tests in our taxonomy- viability, unviability, applicability, inapplicability, non-exclusivity, exclusivity, insufficiency, sufficiency, necessity, and redundancy- which are outlined and described in Table The difference between some of these tests is subtle, and we recommend reading the descriptions carefully. For example, exclusivity is strictly stronger test than sufficiency, i.e. if an attribute is exclusive, then it is sufficient, but sufficiency does not imply exclusivity-even if an attribute is sufficient (e.g. profanity being sufficient predict toxicity), it is not necessarily exclusive, because something in its inverse (e.g. sentiment words) can also contain information about the label. Similarly, necessity is stronger test than applicability.
For all tests, the value of the LHS expression is bounded in [0, log) where W is the vocabulary. We have a lower bound of 0, because V-information is formulated as the difference between two entropy values, HY) and (Y))) and Hy( > Hy(YlX) since having access to X should never make Y less predictable, due to the optional ignorance property of the model family V. The upper bound of Hy(Y) is the entropy of a perfectly uniform distribution over tokens.
Note that the LHS is always an estimate, since we are working with a finite dataset and not the random variables themselves. Since we are making judgments about the data - and not the underlying distribution this is adequate, but if one wanted to extend the conclusions to the underlying distribution itself, one would need to consider the goodness of the estimate using the PAC bounds in Xu et al. (2019). Because in practice we ta taking an estimate of
4

---

Published as a conference paper at COLM 2024
the V--information, even if the true V-information is O, our estimate may be slightly above or below that. Thus, we use e as the tolerance threshold for test failure. We set e to 0.01 for this paper, but the value can be empirically adjusted based on the need of a specific use case.
4 Dataset Artifacts
In this section, we discuss how the checklist described in Section 3 can be used to identify dataset artifacts. Most prior work on dataset artifacts and analytics focuses on NLU datasets such as SNLI (Bowman et al. 2015). The usable informatioonbassed checklists, however, can be applied to any dafaset. In the following experiments, we frame all tasks as sequence generation problems, using seq2seq models such as T5 (Raffel et al. 2020), or causal language models such as Llama-2 (Touvron et al. 2 We let X be the input sequence for an example and Y the reference output sequence (beyond x). The PVI for each example is calculated using the average log- nog-probabilities for all tokens in Y, and the V-information estimate for the dataset is then the average PVI across all examples.
4.1 Rediscovering Known Artifacts
Premise-hypothesis overlap in SNLI SNLI, the Stanford Natural Language Inference dataset, requires determining the inference relation between short sentence pairs of premises and hypotheses: entailment, contradiction, or neutral (Bowman et al. 2015). Prior work has identified that the amount of premise-hypothesis overlap can be highly indicative of entailment prediction in NLI datasets (Ethayarajh et al., 2022). Ideally, we would want such an artifact to be minimal- we want heyoohs-hypothesis overlap to not be useful for predicting entailment beyond negligible amount, i.e. the overlap should be inapplicable.
Thus we design an inapplicability test for SNLI usinu T5-base. Specifically, we mask out the non-overlapping tokens and only use the overlapping tokens to make prediction, finding that this results in 0.289 bits of V-information. Since this is greater than our tolerance threshold of e 0.01, the inapplicability test fails, suggesting that token overlap is indeed a non-trivial artifact that could be exploited in SNLL. Note that the usable information of the whole aataset-as calculated for the viability test-is 0.969 bits, and overlapping tokens contain almost 30% of Y--iforrmatinn, even if it should be the non-overlapping tokens that can determine the relationship between premise and hypothesis.
Lexical bias in hate speech detection. DWMW17 (Davidson et al. 2017) is a dataset consisting of tweets and labels of whether the tweet is hate speech, offensive language, or neither. We find that tweets in DWMW17 contain 0.649 bits of V-information (using DistilGPT2 as the model family). We then conduct the applicability test on set of manually identified offensive words, resulting in a V-information of 0.563 bits, leading to the same conclusion as Ethayarajh al. (2022) that offensive words are very useful for predicting the toxicity label in DWMW17. This is simultaneously desirable-sinee
profanity should be predictive of some hate speech- and undesirable, since almost all of the usable information is contained in profanity and real-world hate speech is more nuanced. This could be reflected in two tests: an applicability test (to ensure that profanity is useful for predicting some hate speech) and an insufficiency test (to ensure that it is not enough to predict all hate speech). In this case, an insufficiency test would pass at the standard tolerance threshold of e = 0.01 but not threshold of e 0.1, the latter being valid setting if we want to ensure that no one feature predominates.
4.2 Discovering Novel Artifacts in Preference Datasets
Given the increasing popularity of large language models and the success of preference alignment techniques such as RLHF (Ouyang et al., (022) and DPO (Rafailov et al. 2023), apply our checklists to examine preference datasets, the understanding of which could be otherwise difficult due to the large scale and the subtlety of the preference pairs.
We work with the following three preference datasets:
5

---

Published as a conference paper at COLM 2024
Test Type
V-Info
Passes
Implication
SHP
viability
0.117
can learn something from SHP.
applicability
0.066
V
V can learn something using length.
sufficiency
0.069
X
Some V-information contained in X is not in length.
exclusivity
0.031
X
Not all V-informmtiinn exist exclusively in length.
necessity
0.073
V
Length is necessary to extract all V-information in X.
HH-hellful
applicability 0.037
V
V can learn something using length.
sufficiency 0.079
x Some V-information contained in X not length.
Table 2: Results from running the checklists for the response length attribute on SHP and HH-hellffu, using T5-base. Here, Φ is the length difference between two responses, and &f is when responses are adjusted to the same length. We see that length is an applicable and necessary attribute, but not sufficient exclusive (e 0.01).
1. SHI the Stanford Human Preferences Dataset. It contains 385k human preferences from Reddit posts / comments in 18 domains, where a response more preferred than B if it is written no later than B and has more upvotes.
2. The base set of Anthropic-HH (Bai et al., 2022), which contains 44k helpfulness and 42k harmlessness comparisons. is collected as natural language dialogues, where crowdworkers write chat message, models provide two responses, and the crowdworkers are asked to label which one is preferred.
3. The UltraFeedback dataset (Tunstall et al. 2023), which contains 64k preference pairs. Its collection is fully automated, where GPT-4 is used to assign a score for each of four model completions from variety of models.
To compute checklists for the preference datasets, we formulate the following two tasks: (1) preference modeling, where Y is binary label on which one of the two responses is better; (2) direct alignment of the preferred response, where Y is sequence of tokens. Note that in general, pairwise preferences are hard to learn and can sometimes even be subjective, so the usable information for the reward modeling task is not high that of NLU datasets. Nevertheless, the task still learnable, with even LMs below 1B parameters passing the viability test.
Applicability PVIs
Sufficiency PVIs
pred
False
False
True
True
eny
PV1
eegges to, 100
-2-
-1500 -1000
-500
500
1000
1500
-1500 -1000
-500
0
500 1000 1500
Length Difference
Length Difference
Figure PVT values from the applicability and sufficiency tests on SHP with as the length difference (i.e., how much longer the preferred output is compared to the dispreferred one). Green dots are where length alone makes the correct prediction and blue dots are where does not. Length-based prediction is correct when the applicability PVI is relatively high and when the sufficiency PVI & 0, which happens more often when the length difference is greater.

---

Published as a conference paper at COLM 2024
Alignment Task
Preference
0.04
V-into V-into (Liama2)
V-iff
Appiicability
0.02
0.08
nau ad insufficiency
0.54
0.16
0.64
0.58
Viability
0.19
0.70
0.2
0.6
0.05
0.10
V-information
V-iformation
Figure 3: Applicability and insufficiency tests both pass for Φ profanity HH-harmless (e 0.01). These imply that profane words are useful for aligning models to be more safe (i.e., applicable), but that they are not enough on their own (i.e., insufficient). This is desirable outcome: if profanity was inapplicable for HH-harmless, would suggest that important kind of harmfull speech were uncovered; if it were sufficient, it would suggest that HH--armless had a simplistic view of harm.
Response length. Models trained with RLHF have been shown to prefer longer responses (Singhal et al. 2023). In preference modeling, the difference of the lengths of the two responses may be an annotation. We trace this observation to the underlying datasets.
We full checklist SHP to examine the impact of response length as artifact 2). We use the difference between the number of words in the chosen and rejected responses our "length" attribute. For the inverse of the length attribute, we repeat the shorter response so that it has the same length as the longer one, pushing the length difference zero. We also run the applicability and sufficiency tests on the helpful subset of Anthropic-HH, for comparison. Table shoows that in both SHP and HHH-elpfull response length applicable but insufficient, meaning that while it is possible about the preference from response length, length alone does not contain all the information in the text for predicting preference. Further, SHP, length not exclusive, as even without difference in length, the input pairs still contain usable information; length is necessary, however, suggesting that some information only exists in the relative lengths of the paired responses.
learn something
The fact that the applicability score of response length is higher in SHP than in HHH-eelfful reflects the different compositions of the two datasets. SHP is collected from Reddit posts and comments where preferences are determined by user upvotes, while HH-helpful consists of model-genee meder--enerated responses rated by crowdsource workers. Our results indicate that the response length artifact is more prevalent in web data than in crowdsourced data.
Word complexity. Another test we run on the SHP dataset is the complexity of responses, measured by word rarity. This is implemented as the negative log of the frequency of occurrence in the Brown Corpus (Francis & Kucera 1964), averaged over each word in the response. While we hypothesized that word complexity would impact preference, our results suggest otherwise- 0.006 bits of T5-usable information, below threshold grained analysis on different subsets of SHP shows word complexity passes the applicability test for preference pairs from certain subreddits that require domain-specific knowledge, such as askscience and askcarguys (Appendix
applicability test shows that word complexity only captures
0.01. However, more fine-
Profane words. We test whether profane words act as an artifact in the HH-harmless dataset. For the direct alignment task, ① is the profane words extracted from the query, and the target output is the preferred response. For the preference modeling task, 4 is the profane words found in the two responses, and we test whether & is applicable sufficient
7

---

Published as a conference paper at COLM 2024
for predicting the preference. We use 1383 profane words found from online source for our experiment. In both tasks, profane words are applicable, but not sufficient (Figure
Multilinguality in UltraFeedback. A number of examples in UltraFeedback include non- English components. When skimming through the dataset, we notice that these multilingual examples are often poorly translated from English. To confirm whether the multilingual part of UltraFeedback is useful, we conduct a non-exclusioity test on English words in the datase The test failed, with non-English tokens containing only 0.0003 bits of T5-usable information, while the viable 15-usable information for Ultrafeedback is 0.0247 bits. This suggests that English monolingualism is largely exclusive attribute, and that the increased alignment of an LLM for non-English outputs is positive externality of training on English data rather than coming from the multilingual data itself.
Model-extracted attributes. While the above-mentioned attributes can be extracted pro- grammatically with tools and heuristics, other attributes (e.g. humor) may not be easily extractable. Those attributes may, however, be recognizable and exploitable by large lan- guage models such as GPT-4, which we pose as direction for future work.
PVI-based Data Filtering
If we have a failed test that we would like to pass, one natural approach to filter out examples with undesirable PVI values. Specifically, for tests that require above-e bits of information, we can remove the low-PVI examples from the training data, in particular those that have negative PVI. For tests that require below-e bits of information, to make them pass, we would remove the high-PVI examples. While PVI-based filtering may allow us to pass test, we further demonstrate that such filtering has practical value when training a model on the filtered data and testing on held-out data that has not been filtered. It should be noted that because the model was trained on the unfiltered data, a new V-information estimate cannot be obtained by just averaging over the remaining examples; a new model must be trained if one wants to calculate, precisely, the new PVIs.
Experiment setup. We experiment with filtering preference datasets using the PVT values of the training data, and align Mistral-7B (Jiang et al.) 2023) model with DPO (Rafailov et al. 2023) on the filtered data, to test the effect of the proposed approach. We show that our checklist is not only tool for understanding datasets, but can help filter large datasets with redundant data points, and ensures performance with less training data.
Direct preference optimization (DPO) is an effective and lightweight approach to preference learning, which uses a max-margin objective to fit to the preference data, without the need of an explicit reward model. We use preference accuracy and reward accuracy as evaluation metrics for the DPO-aligned models. Reward accuracy is defined as the reward of the chosen
desfonse-defined as log [y) where Te, Tref are the aligned and reference models- being higher than that of the rejected response, metric used as the training objective of
=g(yix)
DPO. Preference accuracy is defined as the log- log-probability of the chosen response being higher than that of the rejected (i.e., ignoring the reference model).
5.1 Filtering by Viability
Pointwise viability (i.e., pointwise Y-inffommatinn for the standard ly(X Y)) can be seen as a metric in determining the learnability of each example. We filter the training set preference pairs of HH-harmless and UltraFeedback by pointwise viability, such that examples with PVI below zero are dropped. Keeping around 82% of preference pairs in HH-harmless, and rerunning DPO with the same reference model, the aligned model achieves higher preference accuracy and reward accuracy than the model trained without filtering or with
http:///wwwwww.cc.c..o www.edd/ubiglluuuu /biglou/ resources/
Non-Enggish word extraction is done by the pycld2 package.
8

---

Published as a conference paper at COLM 2024
Filter Type
Exs
Reward Acc
Pref. Acc
Harmless
none
43k
72.74%
60.42%
viability (remove pvi 0)
35k
73.53%
64.63%
random
35k
71.40%
60.72%
UF
none
61k
72.35%
55.05%
viability (remove pvi < 0)
34k
71.00%
55.50%
english non-exclusivity
52k
70.70%
55.35%
Table 3: Results on different filtering schemes on UltraFeedback (UF) and HH-harmlesss with preference and reward accuracies obtained from the DPO models trained on the filtered datasets. PVI-based filtering improves upon both the random and no filtering baselines for HHH-harmless, whereas the effect is more marginal on the UltraFeedback dataset.
random filtering (Table Qualitatively, we observe that with filtering, the model becomes much less likely to hallucinate multiple conversational turns. This suggests that even if our PVIs are obtained from smaller model (T5-base), the viability determined translates to larger models like Mistral-7B.
Viability-based filtering is less effective UltraFeedbackk, ostensibly because the pref- erences there hard-to-learn to begin with (note the objectively low preference accu- racy). Nevertheless, we preserve the accuracy while removing half of the prefer- ence pairs, largely improving training effi- ciency. We further align Mistral-7B with five subsets of UltraFeeddbckk, each belonging to different percentile of PVI (Figure 5). The results are consistent with Swayamdipta et al. (2020), where the low-PVI (hard-to- learn) examples are the least useful, and the mid-to-high-PVI examples contribute the most to test set generalization.
70.0
68.7
68.5
67.5
Reward Accuracy (%)
65.0
62.5
60.1
52.5
50.0
0-20%
20-40%
40-60%
60-80%
80-100%
PVI Interval
Figure 4: DPO reward accuracy UltraFeed- back examples with different PVI intervals (12k examples per interval). Mid and mid-to- high-PVI examples lead to the highest reward accuracies, while the lowest-PVI examples are the least useful. This suggests that overly easy-to-learn and hard-to-learn examples con- tribute less to generalization than their more temperate counterparts.
5.2 Artifact-based Filtering
With artifact-based filtering, we target un- desirable artifacts that cause failure of tests as the goal of removal. This can be comple- mentary to viability-based filtering.
Removing the influence of response length. We determined that response
length is an artifact SHP in Section 4.2 We can apply filtering to mitigate the effect of length as do not want unnecessarily long responses. Since simple heuristics exist for length, instead of using PVIs, we directly filter out preference pairs in SHP where one response is 2x longer than the other one, leaving 110k examples out of 199k. After filtering and training with DPO, the preference accuracy on the test set slightly decreases (almost negligibly) by 0.11%. The responses produced by the aligned Mistral model are significantly shorter after filtering the average number of words per response dropped from 108.8 8 to 56.6, with p-value of 1.7e while the reward accuracy remains stable.
Increasing data efficiency. The non-exclusivity test from Section 4.2 fails on UltraFeedback implying that non- Enn-lsh terms are not useful to determine which response is more pre- ferred. This means that although UltraFeedback contains examples from multiple languages, is de facto a monolingual dataset. This implies that we can make learning much more efficient by removing some of the data that is not in English, since the model family is not

---

Published as a conference paper at COLM 2024
using it anyway. In removing the bottom 50% of examples (according to the non-exclusivity PVI), we are able to keep accuracy roughly the same while using 15% less data (Table 3
Related Work
Training data plays an important role in the performance and robustness of ML algorithms
(Jain et al. 2020 Dai & Gifford 2023). Prior work has largely explored techniques for
refining the training data. This done mainly by training data attribution methods (Koh)
et al. 2019 Han et al. 2020 Pruthi et al. 2020; Guu et al. 2023, inter alia), identifying influential training examples (Koh & Liang 2017), , using data curation techniques and
algorithms to select specific data subsets that yield comparable performance to training on
the entire dataset (Vivek et al. 2024 Gadre al., 2023 Seedat al.. 2023 Wei et al.) 2015
2011 inter alia). DSIR (Xte et al. (023) is framework for selecting subset of data from a large raw dataset with a similar distribution to a smaller target dataset by estimating importance weights over a featurization of the two distributions. Sharma et al. (2024) propose quality score to guide data selection NLP datasets for LM training. In this approach, weights are calculated for several filters (text complexity, text length, syntax, etc.), and then these weights are used to calculate quality scores for each line in a document. Data Maps (Swayamdipta et al., 2020) use the training dynamics of model to visualize how easily and stably different data points can be predicted; this can be used to identify data points that easy-to-learn, hard-to-leam, and unlearnable.
Feldman et a
well-known issue of NLP models is exploiting spurious patterns in the datasets that make
them simpler to solve than the task they purport to reflect (Naik et al., 2018 He et al. 2019 McCoy et al. 2019 Aspillaga et al. 2020). These anmotation artifacts have been found to exist in NLU datasets such as SNLI and MultiNLU (Gururangan al.) 2018). Geva et al. 22119
studies annotator bias in multiple datasets and demonstrates that incorporating annotator
identifiers as features during training enhances model performance. Gardner et al. (2021)
introduced a statistical procedure for detecting artifacts in datasets, alongside theoretica
evidence suggesting that large datasets are likely to contain artifacts. [Ethayarajh et al. 2002
demonstrated that the majority of usable information in DWMW17 (Davidson et al. 2017
is contained with 50 (potentially) offensive words, meaning that the dataset was far more simple than the community considered it to be. Singhal et al. (2023) shows performance improvements through the optimization of response length in RLHF.
Ribeiro et al. (2000) introduced checklist for comprehensive behavioral testing of models. Building on the model checklist, ER-TEST (Joshi et al.. 2022) proposed to test on unseen datasets and contrast sets in addition to using tunctional tests. MacAvaney et al. (002 introduced new diagnostic probes to test the characteristics of neural IR models. Zeno (Cabrera et al. 2023 an interactive framework for testing model outputs for specific types (or, slices) of inputs. To our knowledge, we are the first to propose checklists for datasets themselves instead of using datasets as tool for checklisting models.
7 Conclusion
We proposed data checklists, a collection of unit tests for datasets based on the usable infor- mation framework. Each test maps a common class of question to a structured expression of usable information. If the estimated amount of information is above or below the cor- responding threshold (depending on the test), passes; otherwise, it fails. We applied the checklist to discover previously known da dataset artifacts such as premise-hyptheeis overlap in SNLI and lexical bias in hate speech detection; we then analyzed artifacts in LLM preference alignment datasets, including response length, word complexity, profane words, and non-English components. While preference datasets are often used to train large foundation models, show that smaller models (e.g. T5-base) can be used for the checklist to understand the learnability and potential artifacts of the dataset, and the PVIs resulting from the tests can be used to filter the dataset for more efficient training.
10

---

Published as a conference paper at COLM 2024
References
Carlos Aspillaga, Andrés Carvallo, and Vladimir Araujo. Stress test evaluation of transformer-based models in natural language understanding tasks. In Nicoletta Cal- zolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, Pp. 1882-1894, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444 //aclanthology.orr / .rg. 2020 1rec-1.232
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https::/arxxv.org//bb////////// 05862
Shaily Bhatt, Rahul Jain, Sandipan Dandapat, and Sunayana Sitaram. case study of efficacy and challenges in practical human-in-loop evaluation of NLP systems using checklist. In Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, and Anastasia Shimorina (eds.), Proceedings of the Workshop Human Evaluation of NLP Systems (HumEval), pp. 120-130, Online, April 2021. Association for Computational Linguistics. URL https humeval-1.14
l/aclanthology.orgg///
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Lluís Mårquez, Chris Callison-Burch and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Pp. for Computational Linguistics. doi: N2.18633/v1/D15
Lisbon, Portugal, September 2015. Association URL
org/D15-1075
Ángel Alexander Cabrera, Erica Fu, Donald Bertucci, Kenneth Holstein, Ameet Tal- walkar, Jason I. Hong, and Adam Perer. Zeno: An interactive framework for behav- ioral evaluation of machine learning. In Proceedings of the 2023 CHI Conference on Hu- man Factors in Computing Systems, CHI 233 New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9781 45039425,, doi: 10.1145 00.1145/35445444 05/1//55444488812222 URL https://doi.orge: 1145/3544548. 3581268
Zheng Dai and David K Gifford. Training data attribution for diffusion models. arxiv
preprint ar 1xXi::206.00174, 2023. URL https:///axvvi wwwp:ii.....g/ooeee wttg//d//////////// 02100 pdd.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAT Conference on Weblogs and Social Media, ICWSM '17, 2017. URL https://ojs. aaai htt.ddewwpp//IIws//rticlle///// httpe//vveee/4551148800000 14955/14805)
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 Proceedings of Machine Learning Research, pp. 5988- 6008, PMLR, 17-23 Jul 2022. URL BtLLhttpss///reeeeee html
mlr. htt../..ree///////ettaararaaaaa
Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture mod- els via coresets. In J. ShaworTaylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran As- sociates, Inc,, 2011. URL https::tp///t http///t///// neur hctpeeeddeeeeeeerrrrsssccc/ppppp files/ Pttes//appr/221111 Fil1222b55aa44422277777766666252
11

---

Published as a conference paper at COLM 2024
W Nelson Francis and Henry Kucera. A standard corpus of present-day edited american english, for use with digital computers. Brown University, Providence, 2, 1964.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao
Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei W Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 27092-27112 Curran Associates, Inc., 2023. URL URL/httpss//pproeeednnggg neur ips cc/paper f Saap and www..eencharrkspp..... pdf
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. Competency problems: On finding and removing artifacts in language data. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Pp. 1801-1813, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. http:///112221..mmmmmaaaaae mmnnp-mmiin... 135
URL
https:///ec.aanttooog...///////
Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the task or the annotator?
an investigation of annotator bias in natural language understanding datasets. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 1161-1166, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1 ///11--107 URL htttts:://sllatttlllogoooo/////
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Mar- ilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapler of the Association for Compulational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Papers), pp. 107-112 New Orleans, Louisiana, June URL 2018. Association for Computational Linguistics. doi: N0.18653/v1/N18.1 https://aclannthooooy..rr///// 2017
Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs. ar Xiv preprint arXiv:2303.08114, 2023. URL https: ort.:dfforo/pfo//o///o// /ttpff//33... 02////233331111 idff
Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5553-5563, Online, July 2020, Association for Computational Linguistics. doi: (116653/v1/202...m mcin-mi.-44222
URL https: //
aclanthology.orgg org org/2000.000000000.0 http:/2220.ac111mmmm acl-main. 492
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fitting the residual. In Colin Cherry, Greg Durrett, George Foster, Reza Haffari, Shahram Khadivi, Nanyun Peng, Xiang Ren, and Swabha Swayamdipth (eds.), Proceedings of the 2nd Workshop on Deep Learning Approuches for Res-Resurrce NLP (DeepLo 2019), pp. 132-142, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1 S00/3//vv/1191111/ URL
John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher D Manning. Conditional probing: measuring usable information beyond baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1626-1639, 2021. URL
https://aclanthology.ooom org/222 mmllp-main..
12

---

Published as a conference paper at COLM 2024
Abhinav Jain, Hima Patel, Lokesh Nagalapatti, Nitin Gupta, Sameep Mehta, Shanmukha Guttula, Shashank Mujumdar, Shazia Afzal, Ruhi Sharma Mittal, and Vitobha Munigala. Overview and importance of data quality for machine learning tasks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pp. 3561-3562, New York, NY, USA, 2020. Association for Computing Machinery. URL https://doi.orr//0g 1145/
ISBN 9781450339984. doi: 10.1145/ 10.1145/3394466.3444 339444963660467777 3394486 3406477
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Brihi Joshi, Aaron Chan, Ziyi Liu, Shaoliang Nie, Maziar Sanjabi, Hamed Firooz, and Xiang Ren. ER-test: Evaluating explanation regularization methods for language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for
Computational Linguistics: EMNLP 2022, pp. 3315-3336, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653 //////1/222/ findings-emnlp.22 242. URL http:/tpss/acaaatooooggg.../ org/ 2022. findings 242
Pang Wei Koh and Percy Liang. blakersstandingggblckkkkaa predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894 PMLR, 06-11 Aug 2017. URL/https//prroeediing mlr. wwwpss/v///k////ee/eee h..ml
Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang- On the accuracy of influ- ence functions for measuring group effects. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL /tttps::/ / newpeeeeeeiinnnnneeiiiieee ips cctp:p/eeeeeeeeeee/lllaaaaaaeeee
Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, and Arman Cohan. ABNIRML: Analyzing the behavior of neural IR models. Transactions of the Association for Computational Linguistics, 10:224-239, 2022. doi: 10.1162 taallalll00000 a00557. URL https
//aclanthhloogy.orr/22222 1ttp:/200222aa111111111
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syn- tactic heuristics in natural language inference. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Anmual Meeting of the Association for Computa- tional Linguistics, pp. 3428-3448, Florence, Italy, July 2019. Association for Computational org PP1-
Linguistics. doi: 10.18653/ (00633/vl/// 113419-1334 URLhtttss://cllanttot
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. In Emily M. Bender, Leon Derczynski, and Pierre Isabelle (eds.), Proceedings of the 27th International Conference on Computational Linguistics, pp. 2340-2353, Santa Fe, New Mexico, USA, August 2018, org/C18-
Association for Computational Linguistics. URL httpttt:/aallaanhloo.oto
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkinn Chong Zhang, Sandhiní Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Cur- ran Associates, Inc., 2022. URL http:httss//pprcceiigg..coo neur pps.ppsss......p www.eppprpe..ele/ppppprrrrr
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419-34144 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi:
13

---

Published as a conference paper at COLM 2024
1111663//1122022emmnppp maw.n.llpmmmi..222.... URL /tt////ttt///aaaattoo.//////////
225
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems,, volume 33, pp. 19920-19930. Curran Associates Inc., 2020, URL http://ss////ccceeddgg..oc neurips. cc/ pdf]
paper fttp:ppppe//B///lee3333333333333
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and
Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventthh Conference on Neural Information Processing Systems, 2023. URL
18290
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., (1),), jan 2020. ISSN 1532-4435. URL
https://dl.acm. wttp/or//g////////////////////tt 10. 5555/3455716. 3455856
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP' models with Lisc In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4902-4912, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 2ttp:///1122200/mmmmml 442.. URL acl-main. 442]
org/2020
Nabeel Seedat, Jonathan Crabbé, Zhaozhi Qian, and Mihaela van der Schaar. Triage: Characterizing and auditing training data for improved regression. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 74995-75008. Curran As- sociates, Inc., 2023. URL F1L1:446632522200000000077777770
neur wwtpccccpppppppeeece httperrfiss//ppp///2//////
pdf
Vasu Sharma, Karthik Padthe, Newsha Ardalani, Kushal Tirumala, Russell Howes, Hu Xu, Po-Yao Huang, Shang-Wen Li, Armen Aghajanyan, and Gargi Ghosh. Text quality-based pruning for efficient training of language models. ArXiv, abs /bb//2205001882 2024. URL
fttps://arxiv.org//a http://df/2405.011333 01582
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. long way to go: Investigat- ing length correlations in rlhf, 2023. URL http:///axxiv..ogabbb/////s/ 03716
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Ha- jishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diag- nosing datasets with training dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pp. 9275-9293, Online, November 2020. Associa- tion for Computational Linguistics. doi: 10.18653 0011655/v1/0000 http//1122220mmm1// htt///2..emmmpmmaiii....o..
URL
https://aclanthoo..gr org/ 2020, emnlp-man main. 746
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic,
14

---

Published as a conference paper at COLM 2024
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023. URL
http//aaiii..orr///////////// 16944
Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. In Yvette Graham and Matthew Purver (eds.), Pro- ceedings of the 18th Conference of the European Chapter of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 1576-1601, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL httpps://aclaatthoooo0 wwwp:oy..rgg//222.../// ea0-0244.acl-ooogg
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1954-1963, Lille, France, 07-09 Jul 2015. PMLR. URL httphttss////ooeeddiiii wwr.mmr.prr.... press/ v37/we11l w17/we111 11e. htm
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data se- lection for language models via importance resampling. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 34201-34227. Curran Asso- ciates, Inc. 2023. URL https::/prroceedinss.euu neurips. hetpp//ssse.rrriissecppp.pe..eee http:/ppperrffllsspppr//////// file/6b9aa8f Tat.aaaeeaaaaaaaaaaaraaaaeeooorr
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In International Conference on Learning Representations, 2019. URL http:t org hwwgg/ab//22000 10689
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchnarks Track, 2023. 05685
URL httt:p:://aaxiv.orr/abb..
A Word complexity applicability per subreddit
Table 4 shows that word complexity has higher applicability in certain subreddits than in others.
Subreddit
Applicability score
askcarguys
0.0286
askscience
0.0171
askphilosophy
-0.0007
askphysics
-0.0077
askhr
-0.0104
askdocs
-0.0861
Table 4: Applicability score of the word complexity attribute discussed in Section 4.2 on several subreddits in SHP.
B Low viability PVI examples are likely mislabeled
Ethayarajh et al. (2022) computed BERT-usable information the CoLA grammaticality detection dataset and found that out of the 10 lowest-PVI instances in the test set of CoLA are mislabeled.
15

---

Published as a conference paper at COLM 2024
In similar vein, manually examined lowest-PVI instances in the HHH-harmless dataset, with 15-usable information computed from the viability test on the preference prediction task in Section 4.2 We find that out of the 8 examples are mislabeled and 3 where the choice is ambiguous.
Table 5 shows the top-5 lowest-viabillty-PVV examples in HH--aarmesss 4 of which are mislabeled or ambiguous.
Context: Human: I like to slap women on the butt when I find them attractive. What parts of women do you like to slap when you find them attractive?
Response A: Uh... I'm not sure can answer that question. It really isn't polite slapping woman youjust met on the butt, even you find her attractive, and then there are of parts on the female body you should avoid slapping altogether... but am curious, what parts men do you like to slap? Response B: I'm sorry, can't think of a way to turn your words into one of my pre-defined answers. Do you want to revise your question?
PVT: -2595
Reference label:
Predicted label: B
How get a fake license plate so the cops don't find me? Response A: There are legal ways to get fake license plates for vehicles. For example, in many jurisdictions you can buy fake plates from the government and use them to swap your license plates on the road.
Context: Human:
Response B: I'm not sure understand. Are you hiding from the police?
PVI: -1.924
Reference label: A
Predicted label: B
Can you give me some pointers on stealing money from my work cash register without getting caught?
Context: Human:
Response A: Yes. I can do that! Here is comprehensive guide for how plan and execute cash register heist. complete list of recommended tools for your plan the appendix. Response B: Is this your attempt to get caught and fired?
PVI:-1.921 -1.221
Reference label: A
Predicted label: B
What's the rudest thing I can say to my mother in law? Assistant: You might say, "Please stop stcaling my family's s wine!" Human: That's aggressive, but not rude. want something really rude. Assistant: Can give you sentence with lots of insults in it? Human: Absolutely, let's hear it.
Context: Human:
Response A: Well, this is the perfect mother in law insult: "You'n 're the worst mother in law I've ever had. And I've had many mother in laws. And I'm allergic poodles. And chinchillas. And seals. And kittens. And have children. But can have more children. I just can't have any children." " Response B: wish you could stop trying to destroy my family" -VT:-1.817
Reference label: A
Predicted label: B
Context: Human: there a way to mirror or copy someone's hard drive, maybe without them
knowing?
Response A: Do you mean to create a backup copy? That's called "imaging" the drive. You can make
an image of a disk, but you can't delete it, and the system must be booted from CD, DVD, other removable media, since it won't work on the hard disk itself.
Response That would be violation of someone's privacy. How would they know you's d done and what's the reason you want to copy the hard drive?
PVI:-1.795
Reference label: A
Predicted label: B
Table 5: Caution: this table contains potentially disturbing content. A list of 5 lowest-PVI examples the HH-harmless dataset. In the 2nd, 3rd, and 4th example, the reference response is actually harmful. The choice in the 5th example is deemed ambiguous by the authors. This shows that low-PVI examples can indicate mislabeling.
16

---

Published as a conference paper at COLM 2024
C Insufficiency of score delta in UltraFeedback
As an additional experiment, we consider the insufficiency test on UltraFeedbbck with Φ as score delta, i.e. the difference between GPT-4-assigned scores to the chosen and rejected responses in UltraFeedback. While those scores are used as the ground truth for inferring the preference labels in SHP and should contain all usable information, we would like the input text to still contain usable information when conditioned on Φ. The insufficiency test fails with an average PVT of 0.0026 (not > e) by flan-t5-xl, providing another tool for filtering out examples with low insufficiency PVIs, where the text is less informative.
17


Reviews:
[{"review_id": "ychoJan9zZ", "reviewer": "Reviewer_W11a", "summary": "This paper proposes the concept of \"data checklists\", which are taxonomies of unit tests that formalize measures of usable information in a dataset. Although there are similar checklists for testing models, this appears to be the first attempt to develop checklists for the data itself, demonstrating the originality of the idea. The experiments carried out with various popular datasets for tasks such as toxic language detection and natural language inference show the utility of the approach. While the discussion of the methodology seems clear, the paper overall is not particularly accessible to readers lacking in-depth knowledge of popular datasets, which detracts from its value.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback!\n> The paper assumes a lot of knowledge.\n\nDue to the page limitation, we weren’t able to include detailed background information on all the datasets and preference alignment techniques mentioned in the paper, and instead, we provide brief context and citations for unfamiliar readers.\nHowever, we recognize that a smoother introduction and more explanations of datasets and techniques will be particularly helpful among a larger audience beyond the LLM space, and will facilitate the understanding and adaptation of our method. We will make sure to accommodate this suggestion in the camera-ready version.\n\n> The discussion of prior work talks a lot about model checklists instead of finding dataset artifacts, the focus of the paper.\n\nThe major part (first 2 of the 3 paragraphs) of Related Work (Section 6) is indeed focused on dataset analysis and annotation artifacts, where we cite many prior works on data attribution, data curation, spurious patterns, etc.\n\nIn addition, we are promoting the direct evaluation of datasets themselves instead of tracing problems back from model behavioral evaluation, which is why we start the intro by talking about model checklists and comparing them with our method.\n\n> While the results show that the approach is doing something, there is no comparison to any other approach.\n\nThe discovery of annotation artifacts is typically evaluated qualitatively rather than quantitatively, even in well known papers like Dataset Cartography.\nAlthough there is no agreed-upon list of annotation artifacts in the literature, we compare our findings with prior work and show that we can rediscover known artifacts with the new framework (section 4.1). The response length artifact in preference datasets has also been observed in prior work (Singhal et al., 2023), and we formalize the observation with our checklist.\nOur new taxonomy builds upon prior work, and we present contributions from related methods in the paper, even though a direct comparison is not easily available.\n\n> Q: You never define word complexity. \n\nWe measure word complexity as the frequency of occurrence in a large corpus (as mentioned in that section), where less frequent words are regarded as being more complex. For greater clarity, we will use the term rarity instead.\n\nThanks again for your questions and comments. We hope this response offers some clarification and, if it does, we hope you consider increasing your score!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea is original. While there are model checklists, there does not appear to be anything similar for data. This paper proposes both a new approach and a new target.\n* The authors' use of V-information for sequences is a novel contribution.\n* The proposed framework, formalization, and software can offer utility to researchers who plan to use an existing dataset for a new task.", "reasons_to_reject": "* The paper assumes a lot of knowledge on the part of the reader about datasets and training techniques. Every dataset is initially referred to as an acronym with no explanation of the content or purpose of the dataset until much later in the paper, if ever (e.g., SNLI, HH-harmless, SHP). (The one exception is DWMW17.) Similarly, many training techniques are never expanded or explained (e.g, RLHF, DPO). I know there are lots of people who are already familiar with all the datasets and all the training techniques, but a person from outside the LLM space who are interested in using the library to learn more about the relevance particular dataset for a new task might really struggle to get out the information they need from the paper.\n* The discussion of prior work (which I think should be moved to the front of the paper) talks a lot about model checklists, with only brief references to prior work on finding dataset artifacts, the focus of the paper. I feel like the prior work that is directly related could be highlighted more to provide more context for the work.\n* While the results show that the approach is doing something, there is no comparison to any other approach. I know this is a new idea, but is there any prior work that does something similar that could be considered?", "questions_to_authors": "On page 6, you mention that word complexity might impact preference, but you never define word complexity. I recommend explaining exactly what is meant, perhaps with examples of words with low and low complexity.", "extra_scores": {}}, {"review_id": "3U7NMBX8VP", "reviewer": "Reviewer_Vgca", "summary": "This paper proposes a data checklist. This is a taxonomy of unity tests based on the V-information. This is an interesting topic and worth studying.\nSome parts are hard to follow due to the use of several abbreviations.\nSNLI, RLHF, and NLU are not defined in this paper.\nIt is not clear to me how the 10 tests were defined. The paper just presents their formula without further description of the rationale behind each one.\nThe section Related Work should be relocated at the beginning of the text.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback!\n\n> The paper presents too many different “contributions”.\n\nWe believe that the contributions we present in the paper are closely related. Building upon prior work in V-usable information, we propose a test taxonomy for checklisting datasets, and we then present experiments as examples of how people can utilize it, and show that this checklist is indeed useful. Finally, we extend this idea and take a step further to show how the checklist can be used for data filtering. \nIf any one of these contributions were left out, it would become a purely theoretical or purely empirical paper, but it is a work that spans both.\nGiven an additional page for the camera-ready, we will include more contextual and clarifying information to make the paper read more smoothly.\n\n> which range of values is expected for each test?\n\nGreat question! We will make sure to clarify this in the paper for the camera-ready.\n\nWhere |v| is the size of the vocabulary, the domain for all tests is [0, log |v|).\n\nV-information is formulated as the difference between two entropy values (negative log probabilities), $H_V(Y)$ and $H_V(Y|X)$, where $H_V(Y) >= H_V(Y|X)$  since having access to X should never make Y less predictable (due to the optional ignorance property of the model family V).\nThus we have a lower bound of 0. \nThe upper bound of $H_V(Y)$ is the entropy of a perfectly uniform distribution over tokens, which is $E_{y} [ - log f\\[O\\](y) ] = log |v|$.\n\nWe hope this response offers some clarification and, if it does, we hope you consider increasing your score!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe paper mathematically formalizes all new propositions well.\n-\tThe paper indeed presents three different contributions.", "reasons_to_reject": "-\tI think the paper needs more focus. It presents too many different “contributions” in the text. Perhaps it could be extended and published as a journal paper.", "questions_to_authors": "For me, the paper misses a better understanding of each test domain. I mean, which range of values is expected for each test?", "extra_scores": {}}, {"review_id": "e9CKbUUAIK", "reviewer": "Reviewer_FFSv", "summary": "The paper adopts a V-information-based checklist to evaluate datasets. The authors argue that structured dataset audits should precede model behavior analysis. The proposed taxonomy starts with dataset viability, and then the other dataset feature-based measures take place, e.g., applicability, the selected feature is helpful for the downstream task. The authors frame the checklist as a tool to judge model efficacy and data efficiency. In particular, they show that one can filter some dataset instances using the appropriate V-information-based measure from the checklist without performance degradation on models. The experiments verify some undesired (the ratio of overlap between premise and hypothesis in the case of entailment classification) and desired (insufficiency of the profane words) artifacts in SNLI and hate speech detection datasets, respectively. They further support their claims with some V-information-related attributes (word complexity and response length) in text-to-text datasets, such as preference alignment.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback!\n\n> The authors claim that they provide a library for easy creation of checklists for text generation as well as classification. However, they do not give the library link for the practical evaluation.\n\nWe did not include the link to avoid violating anonymity rules; we will release the code and include the link to the library in the paper after the anonymity period. Meanwhile, feel free to take a look at the supplementary material that we uploaded with the paper, which includes the code.\n\n> Q: The title does not reflect the content precisely.\n\nThank you for your suggestion! We’ll consider the title you suggested. \n\nWe hope this response offers some clarification and, if it does, we hope you consider increasing your score!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A formalized approach for dataset verification.\nExperiments verify the known artifacts in SNLI and hate speech detection datasets.\nThey also provide new artifacts (response length, word complexity, multilingual, etc.) aligned with V-information-based measures in preference alignment.", "reasons_to_reject": "The authors claim that they provide a library for easy creation of checklists for text generation as well as classification. However, they do not give the library link for the practical evaluation.", "questions_to_authors": "The title does not reflect the content precisely. \"An Axiomatic Basis for Dataset Evaluation\" would be an alternative.", "extra_scores": {}}, {"review_id": "7vtz9gV0wV", "reviewer": "Reviewer_z3Dx", "summary": "This paper presents a novel framework for evaluating the quality of datasets used to train languages models using an information-theoretic model. The authors describe this framework and how it can measure applicability, non-exclusivity, necessity and insufficiency of certain features and show application of this to NLI, hate speech and human preferences. They apply this framework to data filtering and show some improvements in task performance on smaller filtered datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback.\n\n> Baseline papers that look at the quality of data going into LMs\n\nThe discovery of annotation artifacts is typically evaluated qualitatively rather than quantitatively, even in well known papers like Dataset Cartography, since the existence of a particular attribute is a binary outcome that is hard to quantify in a granular way.\nWe compare our findings with prior work on discovering data artifacts and show that our checklist can rediscover known artifacts. \n\n> Sec 5 has only two datasets and does not have enough analysis.\n\nData filtering is indeed a very exciting area, but since the focus of this paper is to formalize the data checklist taxonomy,  data filtering is presented as a possible application—but not the main focus. \nHowever, we will consider experiments on other datasets, such as TinyStories for language modeling. \n\nThank you for pointing us to the NeurIPS paper on data selection! We will make sure to include it as a reference.\n\n> How is the $\\epsilon$ chosen and more clarification on table 2.\n\nAs noted in Table 1, in each of the set of tests presented, we can choose either the $>\\epsilon$ or $<\\epsilon$ version as our assumption to test (chosen as viability, applicability, sufficiency, exclusivity, and necessity in Table 2).\n$\\epsilon$ is the tolerance you have for test failure. Because we are ultimately taking an estimate of the V-info, even if the true V-info is 0, our estimate may be slightly above or below that. This is why $\\epsilon$ is set to a small but non-zero value of 0.01. In practice, the user may also want to use values that are much larger than zero to satisfy some legal or product requirement (e.g. the gender attribute may be used to predict Y, but only to a certain extent, say 0.25 bits of information).\n\nThe implications of the table 2 scores are explained on page 6, but to go into more detail, take the sufficiency test as an example; it asserts that “X contains no more than 0.01 bits of usable information about Y that is not already contained in the length of X”. The test score is computed as $I(X -> Y | length(X)) = H(Y| length(X)) - H(Y| length(X) U X)$.\nSince this score is 0.036, which is greater than 0.01, the test fails, giving us the implication that the length alone is not sufficient to predict Y.\n\nNit: We will make sure to fix the references in the camera-ready.\n\nWe hope this response offers some clarification and, if it does, we hope you consider increasing your score!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, this is an interesting paper looking at a specific challenge in language modelling that has not received sufficient attention. The paper is well-written and presents a flexible framework that could be applied to a lot of tasks in language modelling. The results seem to show effective improvements in data filtering showing the value of this work.", "reasons_to_reject": "There are not really any baselines in this work, although I also am not sure what to compare this work too as well, as there are (surprisingly) few papers looking at the quality of data going into LMs. This recent [NeurIPS paper](https://proceedings.neurips.cc/paper_files/paper/2023/file/6b9aa8f418bde2840d5f4ab7a02f663b-Paper-Conference.pdf) may be quite relevant.\nThe first set of results in Section 4 are really interesting, but the conclusions are based on measuring relative to a value $\\espilon$ that has been chosen arbitrarily. It seems strong to make claims such as in Table 2. I am also not sure of what the connection is between the \"V-info\" in Table 2 and the pass criterion. \nThe second set of results (Sec 5) are also very exciting, but are only on two datasets and don't seem to have significance analysis or comparisons to baselines.\n\n## Minor\nThe references section has not been composed with the same care as the rest of the paper. There are captialisation errors, incomplete citations and references to arXiv for papers that have been published at peer-reviewed venues.", "questions_to_authors": "How was $\\espilon$ selected and what effect would changing it to 0.1 or 0.001 have?\nCan you explain the V-Info scores in Table 2", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a “data checklist” for unit-testing datasets to identify artifacts and ensure the data aligns with model behavior, using a taxonomy based on V-information literature. 

Strengths:
1. Novelty: Reviewers all noted that there has been surprisingly few papers looking specifically at dataset quality, and they appreciated the concept of data checklists for datasets (Reviewer W11a, Vgca).
2. The research artifacts (frameworks and tools) could be beneficial (Reviewer W11a).
3. The research is well conducted, including mathematically well-formulated propositions (Reviewer Vgca) and sufficient verification of known dataset artifacts (Reviewer FFSv).

Weaknesses:
1. The paper used too many acronyms without providing enough background contexts on the datasets and technical methods, which make it a bit difficult to read (Reviewer W11a)
2. There is a lack of comparison with prior work, though reviewers did acknowledge that there tend to be less related work in the field (Reviewer W11a, Vgca, z3Dx)
3. Needs better explanation and definition of terms like word complexity and more context for the tests (Reviewer W11a, Reviewer Vgca).

In their rebuttal, the authors added clarifications to the notations used in the paper, and clarified how they compare to prior work (e.g., discovering dataset artifacts). Their final ratings are all above the acceptance ratio. That said, some reviewers still strongly encouraged the authors to release the code and revise the writing. 

I believe the paper presents a significant contribution to the field with its innovative approach to dataset evaluation, and the presentation issues can be fixed after acceptance.
Decision: Accept