Zero-Shot Data Maps. Efficient Dataset Cartography Without Model Training
Authors: Angelo Basile
Marc Franco-Salvador
Paolo Rosso
Venue: EMNLP 2023 Findings (2023)
Keywords: data maps
bi-encoders
zero-shot
training dynamics
TL;DR: Zero-shot models can make data maps too.

Abstract: Data Maps (Swayamdipta, et al. 2020)  have emerged as a powerful tool for diagnosing large annotated datasets. Given a model fitted on a dataset, these maps show each data instance from the dataset in a 2-dimensional space defined by a) the model's confidence in the true class and b) the variability of this confidence. In previous work, confidence and variability are usually computed using training dynamics, which requires the fitting of a strong model to the dataset. In this paper, we introduce a novel approach: Zero-Shot Data Maps based on fast bi-encoder networks. For each data point, confidence on the true label and variability are computed over the members of an ensemble of zero-shot models constructed with different --- but semantically equivalent --- label descriptions, i.e., textual representations of each class in a given label space.
 We conduct a comparative analysis of maps compiled using traditional training dynamics and our proposed zero-shot models across various datasets. Our findings reveal that Zero-Shot Data Maps generally match those produced by the traditional method while delivering up to a 14x speedup. The code is available  [here](https://github.com/symanto-research/zeroshot-cartography).

Zero-Shot Data Maps.
Efficient Dataset Cartography Without Model Training
Angelo Basile
Marc Franco-Salvador
angelo. bas i e symanto, com Universitat Politècnica de València Symanto Research
marc. francol soro@symmn symanto. com Symanto Research
Paolo Rosso
prossoêdsic. upv es PRHLT Research Center Universitat Politècnica de València
Abstract
datasets into black boxes (Paullada et al., 2020). This division of labour, while economically advan- tageous in terms of time and resources, hinders thorough comprehension of the data. Dataset Car- tography using training dynamics (Swayamdipta et al., 2020) offers partial solution to this prob- lem as it can help interpreting annotated corpora with respect to model. Specifically, data maps reveal three loose groups of instances within an annotated dataset: easy-to-learn, ambiguous and instances; the latter often correspond to annotation errors. Training models based on samples from these regions can lead to better models with improved generalization. While car- tography based on training dynamics can generally compile high quality maps, it does so at the cost of training large model on the target dataset.
Data Maps (Swayamdipta et al., 2020) have emerged as a powerful tool diagnosing large annotated datasets. Given model fitted dataset, these maps show each data instance from the dataset defined by the model's 's confidence in the true class and b) the variability of this confidence. In previous work, confidence and variability are usually computed using training dynamics, which requires the fitting of strong model to the dataset. In this paper, we introduce novel approach: Zero-Shot Data Maps based on fast bi-encoder networks. confidence the true label and variability are computed over the members of an ensemble of zero-shot models constructed with different label
a 2-dimensional space
each data point,
but semantically equivalent descriptions, i.c., textual representations of each class in given label space. We conduct a comparative analysis of maps compiled using traditional training dynamics and our proposed zero-shot models across various datasets. Our findings reveal that Zero-Shot Data Maps generally match those produced by the traditional method while delivering up to 14x speedup. The code is available https: / com/ symanto- research/
In this work, we introduce novel, fast method for drawing Data Maps using zero-shot models. This approach capitalizes the sensitivity of zero- shot models label description framing, offering more efficient and resource- friunddl alternative to the original training dynamics method. Our ap- proach allows for the generation of data maps with- out the requirement of extensive computational re- sources for the training of robust model on a large dataset. When evaluated on data selection and error detection tasks, our experimental results show that zero-shot map coordinates as coordinates obtained through training dynamic maps, while requiring fraction of the time to com- pute. Zero-shot data maps speed up the analysis of large datasets, removing the need for model train- ing; they are handy tool for industry practitioners managing daily unique datasets.
1 Introduction
useful
Practices like annotation crowdsourcing and large- scale web scraping have enabled NLP researchers to create large annotated corpora (Snow et al., 2008; Brown et al., 2020). Concurrently, this growth in data volume has led to an increased preva- lence of label noise and annotation artifacts within datasets (Abad and Moschitti, 2016; Gururangan al., 2018). Further complicating matters, the widening gap between dataset annotators and end- users - researchers and practitioners who use the datasets to train models - often transforms these
Research Questions In this study, we aim to an- swer the following research questions (RQ):
RQ1: Can bi-encoder models serve as the ba-

---

sis for lightweight, low-resource variant of data cartography? In particular, can map co- ordinates obtained through training dynamics be approximated by zero-shot models?
A bi-encoder model operates two-step process when given input texts and label descriptions: first, generates embeddings for both the documents and label descriptions, and secondly, uses these similarity scores as classification function.
RQ2: Can the confidence metrics from en- sembled zero-shot models serve as a reliable indicator for finding annotation errors?
More formally, given corpus of N documents and task with with K classes each with corre- sponding label description 4 bi-encoders firstly independently embed the input texts and the label descriptions. We define the document embeddings as X E IR Nxd and the label description embed- dings as I. RKxd where d' is the dimension of the bi-encoder embedding model E(-). Next, sim- ilarity score is computed for each document and label description pair: this is done using sym- metric function s(;;,) like cosine similarity dot product. Finally, the class associated with the label yielding the highest similarity with the input text is chosen as the prediction:
Contributions We introduce novel method for compiling data maps up 14x faster. We compare our method against training dynamics-based maps, Furthermore, we show that zero-shot maps can be used for automatic annotation error detection. Fi- nally, we will provide an easy-to-use implementa- tion of our method for enabling NLP practitioners to inspect new datasets.
2 Data Maps with Zero-Shot Models
In this section, we first explain how our chosen zero-shot architecture works and why we chose it over other options (Section 2.1). We then proceed illustrate the sensitivity of zero-shot models to label descriptions (Section 2.2). Finally, we de- scribe how we can use this feature to build Data Maps with zero-shot models, thus avoiding model training (Section 2.3).
j = argmax, (F(()), F(4))
(1)
The application of softmax transformation the array of similarities, turns the bi-encoder model into probabilistic classifiers. These transformed similarities create a probability distribution over the labels:
2.1 Zero-Shot Classification with Bi-Encoder Models
P(yl(z) (())) (2)
Recent work in NLP (Radford et al., 2019) have shown that large pre-trained language models (LLMs) can effectively be used in a zero-shot setup, meaning that they require no data to solve a task. Among the different architectures that have been shown capable of turning LLMs into zero-shot clas- sifiers (Radford et al., 2019; Yin et al., 2019; Schick and Schüizze 2021), we focus on the bi-encoder (or Dual Encoder) class of models (Reimers and Gurevych, 2019; Mueller et al., 2022) due their efficiency during inference. Bi-encoder models work as classifiers thanks to label descriptions, which we define as a natural language represen- tation of each class in the label space. Label de- scriptions are essentially the sentence- embeeddng counterpart to the use of task descriptions with pre-trained language models (Radford et al., 2019). For example, consider binary classification task identify spam emails. Here, the label description for the spam class might be This email contains un- solicited content while for the non- non-spam class, could be This email includes relevant information.
In this way, the model provides not just pre- diction but also a probability distribution over all labels, offering an interpretation of the model's confidence for each label.
We opt for the bi-encoder architecture because of its inference efficiency. In particular, bi-encoders can independently model input and labels, meaning that only single pass over the input documents is necessary, regardless of the number of classes or label descriptions per class. contrast, cross- encoder Natural Language Inference (NLI) models (Yin et al., 2019), a popular architecture for zero- shot classification, implies K passes over the input data for each class & within the label space. When defining multiple label descriptions I. per class to create source of variation in predictions do in this work N L K passes would be required, iterating for each class and each la- bel description. In contrast, Mueller et al. (2022) demonstrate that bi-encoders can achieve quality similar to that of cross cross- encoder models while being

---

correctness
variability
variability
Figure 1: Data Maps for the training split (limited to 25k examples) of the SS72 dataset, computed using training dynamics (left) and zero-shot proposed method (right) with sentence-15-large (Ni al., 2022).
up to 20 times faster terms processed tokens per second.
model's performance, as shown Table 1.
For each task, we first manually craft a large set of different label descriptions; next, we gener- ate predictions for all the possible combinations of these label descriptions. More formally, let assume have total of T. label descriptions per class k. For our input text we compute an embed- ding æ. For each label description in class k, we calculate an embedding, denoted as lc,i. Next, we apply a symmetric similarity function f to deter- mine the similarity between x and each lc,i. result- ing in array of similarities Sg for each label description:
2.2
Sensitivity to Label Description Phrasing
LABEL DESCRIPTION
F1
a) Just bad!
89.33
All in all, the movie awesome.
Just terrible!
b)
68.71
All in all, the film was great.
mean min max
78.55
24.79
89.33
Sk J(x, (xx,4x))) 4x.))
Table I: Top: a) the best performing combination of label descriptions (red and green for the negative and positive class respectively) on the SST dataset and b) semantically similar, but much worse, set of label descriptions. Bottom: Mean, minimum and max score across set 729 label description combinations for the SS72 dataset.
Hence, for each embedded input text and each class k we generate array of L predictions (one for each label description). This leads to N C predictions for each input text, where is the number of classes and the number different label descriptions per class. In Figure we show the distribution of macro FI scores computed using large set of different label deseriptions' across several datasets: average, performances range between state-of-the-art and purely random.
LLM-based zero-shot models notably sensi- tive the phrasing and structure of the prompts used for specific tasks (Shin al., 2020; Lu al., 2022). Label descriptions suffer from the same problem. To illustrate this, let us consider the case sentiment analysis task with positive and negative the class labels. The phrasing of label descriptions can vary within the same class. For instance, positive tive descriptions could be All in all, the movie was awesome. or All in all, the film was great., while negative descriptions could be Just bad! or Just terrible!. Despite conveying the same meaning, the specific wording of these descriptions can have a substantial impact on the
2.3 Zero-Shot Data Maps
Considering that bi-encoder classifiers a) deliver efficient inference time regardless of the number of labels, and b) exhibit performance variations based on the phrasing of label descriptions, we can trans-
late the concept of training dynamics to zero-shot maps. More precisely, we compute confidence and variability across the predictions of Z ensemble
'A detailed overview of the label descriptions can he found in Table 8 in the Appendix.

---

among the ensemble members about the correct label for an instance.
08
Finally, correctness simply tracks the number times a model predicted the gold label. In Figure we show an example of data maps compiled using both the training dynamics method and our novel zero-shot approach.
Experimental Setup
We first conduct visual evaluation of the quality of zero-shot data maps, comparing them with the maps obtained through training dynamics. Next, follow the original data selection experiments and train set of models on sub-samples of three datasets. Finally, we evaluate zero-shot maps as error detectors and compare run time and resource usage against training dynamics maps.
Figure 2: Zero-shot Fl performance of difference combinations of label descriptions for datasets.
members of a zero-shot model instead of a series of training epochs. Intuitively, we can view zero-shot maps the equivalent of time-unrolled version of training dynamics map: we take multiple pre- dictions all once from various zero-shot models, each prompted with a different label description, as opposed to gathering predictions from different epochs of the same model. Similar to the variable quality of predictions across training epochs, which generally improve as more epochs are completed, the performance of zero-shot models can also vary substantially depending the label description, as we discuss in Section 2.2: this source of variation serves the basis for computing confidence and variability.
3.1 Data Selection
For the data selection experiments, we first gen- erate maps using both training dynamics and our proposed zero-shot method; for consistency, we use the same model - sentence-t5-large (Ni et al., 2022) - for both approaches. Next, following the existing literature (Swayamdipta et al., 2020), define eight distinct regions based all pos- sible combinations of confidence, correctness and variability. These regions include:
Easy-to-learn:
These are instances that the models consistently predict with high confi- dence. These instances exhibit both high con- fidence and low variability.
The average confidence of a zero-shot model defined by the embeddings of a specific set of label descriptions 0₂ over an instance I; with true label n is defined as the average probability assigned by the ensemble to label yf across all the members Z of the ensemble:
These are instances that the models find challenging. These instances have low confidence and low variability.
Hard-to-learn:
Ambiguous: These instances exhibit high variability and differing levels of confidence. They usually constitute the majority of the instances dataset.
1 Z (a)
14 7
Σ 2=1
(3/ (x;)
Similarly, define variability as the standard deviation of the model's probabilities across the ensemble members:
High confidence: Instances where the models, regardless of variability, exhibit consistently high confidence.
9. L (po (v) 22)
High correctness: Instances that consis- tently correctly predicted across different la- bel descriptions or training epochs.
This measure quantifies the spread of model's probabilities across the ensemble, giving an indi- cation of the degree agreement or disagreement
Low correctness: Instances that are frequently misclassified.

---

Low variability: Instances that yield consis- tent model predictions.
(Maas al., 2011), SST2 (Socher al., 2013) and Cola (Warstadt et al., 2019). Additionally, we plot maps for these datasets together with the Yelp (Zhang et al., 2015) and AG News (Gulli, 2005) datasets. We test automatic error detection using two datasets: a version of IMDB which has been re- annotated using both crowdworkers and automatic methods (Northcutt et al., 2021) and the SST2 dataset where we simulated errors by randomly flipping 5% of the gold labels. Table highlights the key facts about these datasets.
Random: a baseline that randomly selects in- stances from all over the map
Given the map of the entire training set, we order the instances based our strategy of interest (e.g.. hard-to-learn, easy-to-learn, ambiguous, etc.), and then sample from this ordered list until we acquire 33% of the original datasct. In some cases, class might not be represented in the sampled split: we avoid this by including four random instances from each class in the subset. Given a 33% sample of the original data, we train a classifier using exclu- sively such sample and evaluate its performance on the test split. For the zero-shot maps, we use the sentence-t5-large model as frozen encoder and trained a logistic regression model on top.
4 Results
4.1 Visual Comparison
We visually compare zero-shot and training dy- namics maps, revealing a good similarity between them. As can be seen in Figure I, while the training dynamies map exhibits more pronounced spikes, re- flecting distinct training epochs, the zero-shot maps present a smoother profile. This smoothness stems from the larger number of label description combi- nations used in the zero-shot setting, in contrast to the few epochs used in the training dynamics. Fig- ure provides view into the relationship between dataset tasks and the performance of the zero-shot model. For tasks where the zero-shot model yields strong results, such as Yelp Polarity and Ag News, the plotted curve demonstrates a distinct bell shape, which closely mirrors the distribution observed in the training dynamics map. For datasets like Cola, the distribution, although still bell-shaped, shows larger concentration of instances in the lower half of the graph, indicating subpar performance of the zero-shot model on this particular task. These observations suggest that the model is highly sensi- tive to the nature of the task at hand.
3.2 Error Detection
A plausible interpretation for instances populating the low-confidence and low-varialee variability map area might be to consider them as annotation errors. The interpretation is based on the idea that if a model persistently mispredicts the gold label as the most probable label for given instance, it is likely that the gold label itself is incorrect. We test if the confi- dence coordinates provided by zero-shot data maps can serve as features for an automatic error detec- tion system and compare with training dynamics data maps as a baseline. In their comprehensive study of automatic error detection, Klie et al. (2023) demonstrated that the confidence feature from train- ing dynamics maps is as robust error detection baseline, performing well across various tasks and datasets.
We test simple linear model on the semi- automatically validated test subset of the IMDB dataset (Northcutt et al., 2021), using confidence as defining features and flipped gold labels as target. Specifically, we train logistic regression model using the scikit-learn implementation (Pedregosa et al., 2011 1) the train split of IMDB: we sort the instances by decreasing confidence, sample 300 instances per class and flip the gold labels of the first 150 instances, under the assumption that in- stances with high confidence are probably correctly labelled.
4.2 Data Selection
For the data selection experiments, we trained mod- els using data samples selected based on the coor- dinates from both zero-shot and training dynamics maps. The results in Table show that zero-shot maps and training dynamics generally offer compa- rable performance for data selection, though there are some notable exceptions. For instance, the ambiguous category in the Cola dataset shows marked difference, which we hypothesize may be due to the inherent complexities of the Cola cor- pus and limitations in the dual-encoder embedder's ability to assess sentence grammaticality. As such, while zhro-shot maps provide a resource-efficienr
3.3 Datasets
For our data selection experiments, we select commonly used English labelled corpora: IMDB

---

TASK
CLASSES
Ag news (Gulli, 2005)
lopic classification
Business Sei/Tech Spurts World
120,000
7,600
Cola (Warstadr al., 2019)
linguistic acceptability
acceptable unacceptable negative positive
8,551
1,063
1,043
IMDB (Maas et al., 2011)
sentiment
25,000
25 25,000
SST2 (Socher et al., 2013)
sentiment
negative positive
67,349
1,821
872
Yelp Polarity (Zhang et al., 2015)
sentiment
negative positive
560,000
38,000
Table 2:
Dataset Overview. All the datasets are in English. For the data selection experiments, we use the validation
split for Cola and SST2 and the test split for the rest.
correctress
0.c-
correctnoss
conhdence
0.00
625-
0.4
22
valisbiity
verisbilit
correctness
correctness
condeence
672
confdeec
279
0.50
250
varisbilty
wariabity
(d)
Figure 3: An overview of Zero-shot and Training Dynamics Maps. Top: Cola data maps generated using zero-shot models (left) and training dynamics (right). Bottom: zero-shot maps for the Yelp (left) and Ag News (right) datasets.
COLA
IMDB
SST2
alternative, they should not be considered a com- plete substitute for training dynamics, particularly for complex tasks.
100%
92.55
94.04
TD
TD
TD
randem
36.22
36.22
92.10
93.00
93.00
53% train
ambiguous
44.04
37.33
92.22
92.15
92.54
93.35
We consistently observe significant improve- ments over the random baseline data selection for both zero-shot and training dynamics methods. In- terestingly, for datasets that are annotated by hu- mans, such as Cola and SST2, the advantage over random selection is more pronounced. This may be due the less systematic labeling of these datasets, as they are annotated by humans, compared datasets like IMDB which are compiled automati- cally using proxies like star ratings. These findings suggest that the data selection approach using data
to learn
34.13
36.41
92.24
92.25
92.43
92.43
hard to Jearn
34.43
36.01
92.23
92.24
92.43
92.43
high-confidence
25.12
33.38
92.89
high hoghhccosss
34.42
49.43
33.77
92.21
92.89
90.37
low-corretnnes
39.40
46.09
33.53
92.14
93.23
91.63
low-variability
34.10
36.51
92.24
92.25
92.43
92.43
Table 3: Macro-averaged Fl scores from Training Dy- namics (TD) versus Zero-Shot (ZS) generated maps. For each dataset, the best results for both methods are bold font.

---

maps is particularly effective when dealing with datasets that exhibit high degree of labeling com- plexity.
TIME
RAM
Zero-Shot (CPU)
28 min
10.6Gb
Zero-Shot (GPU)
min
2.2Gb
4.3 Error Detection
TD (CPU)
9 hr 5 min 16.4Gb
TD (GPU)
14 min 6.6Gb
The automatic error detection results shown are pre- sented in Table 4. Zero-shot maps even outperform Training Dynamics in terms of average precision, showing that this method has potential as a basis for an automatic error detection system.
Table 5: Comparison of time and RAM usage for Zero- Shot and Training Dynamics (TD) on both CPU and GPU for dataset with 10.000 instances using batch size of 8 and static padding to 512 tokens. RAM is VRAM for GPU experiments and system RAM for the CPU experiments.
MODEL
IMDB
SST2
Training Dynamics Zero-Shot
30.2
49.0
34.7
57.0
more efficient, lightweight method for construct- ing data maps (RQ1). Our experimental outcomes, as outlined in Sections 4.1 and 4.2, suggest that zero-shot data maps dataset when the task can be accurately represented using label descriptions only. Typically, these tasks include most text classification tasks, such as senti- ment analysis and topic classification.
Table 4: Average Precision scores for the automatic error detection systems on two datasets: the semi- automatically cleaned IMDB dataset and the SST2 dataset where errоrs were simulated by randomly flip- ping 5% of the gold labels.
precisely characterize
Figure 4a presents zero-shot map of the IMDB dataset's test split, where instances identified an- notation errors by Northeutt al. (2021) are high- lighted in red, large proportion of instances with incorrect gold labels clusters towards the lower half of the map. This clustering implics that the con- fidence of ensemble zero-shot models can be an effective indicator for identifying error instances, Similarly, Figure 4b shows that the average confi- dence assigned to instances with incorrect annota- tions is generally lower compared to instances with correct labels.
Secondly, we focused on automatic error detection (RQ2) with an emphasis on adhering to the inter- pretations of the data maps regions as provided by Swayamdipta et al. (2020), i.e., assuming that hard-to-leamt instances are likely to be annotation errors. Our findings in Section 4.3 provide evidence that the confidence scores derived from zero-shot classifiers can indeed serve as a reliable signal for constructing error detection systems. These results suggest that zero-shot maps can be a good alterna- tive to training dynamics maps.
For larger academic datasets, which often re- quire substantial investment in terms of both time and financial resources, the cost associated with training supervised model could well be justified. In applied settings, particularly in industry, datasets constantly being created and updated. In these scenarios, we recommend the use of zero-shot data maps as they offer swift, efficient method for understanding data, while still delivering perfor- mance on par with training dynamics maps. With recent advancements in NLP technology, products like ChatGPT (OpenAi, 2022; OpenAI, 2023) have made it much easier to label new data for wide range of applications. These models have, in some instances, even begun replace the need for hu- man annotators and crowdsourcing (Gilardi et al., 2023; He et al., 2023). Despite the impressive ca- pabilities of these models a zero-shot setting, they still commonly make mistakes. Consequently,
4.4 Computational Requirements
The main advantage of our method is that it by- passes model training. This speedup is further in- creased by the zero-shot architecture choice, i.e., the bi-encoder network. A comparison of the com- putation time and resources used for generating training dynamics and zero-shot maps reveals that zero-shot maps are generally much faster to com- pute, as illustrated in Table 5. The absence of a training phase allows for a different use of re- sources: given the same GPU, larger often better encoding models can be used in the creation of zero-shot maps as there is requirement for the backward pass typically needed in training.
4.5 Discussion
this study, our goal was to address two key re- search questions. Firstly, we aimed to create a

---

soufdence
anaer hulaa
error
025-
varobilty
conf dence
(a)
Figure 4: a): A zero-shot data map of the IMDB test split. We show in red the annotation errors. b): the confidence distribution for the test split IMDB instances with a correct gold label (error False) and incorrectly labeled ones (error True)
labeled test sets continue to a critical require- ment for their proper evaluation. This fact under- scores the need for fast and accurate data mapping techniques that can be used to identify potential (automatic) annotation errors. This context further highlights the value of the zero-shot data maps pro- posed in this study.
fication by using label descriptions encoded with shallow features for zero- and few-shot classifica- tion. Recently, Müller et al. (2022) have shown how modern approaches using dense bi-encoders can outperform more resource-intensive cross-encoder models. For comprehensive review of automatic annotation error detection, et al. (2023).
refer Klie
5 Related Work
Conclusion
Recently, several lines of research have presented findings related to the concept of Dataset Cartog- raphy. Sanh et al. (2020) demonstrated how weak models, as opposed strong ones, recreate sim- ilar data maps by leveraging annotation artifacts and biases. On the subject of instance difficulty, Ethayarajh et al. (2022) proposed V-usable infor- mation, a generalization of Shannon information (Shannon, 1948), which provides an aggregate mea- sure of data difficulty in relation to a model; the pointwise V Information, an extension of V, has been shown to correlate strongly with data map coordinates, and offers the additional benefit of ex- plaining why an instance can be challenging for model. Data maps have been recently applied to variety of tasks such as syntactic parsing (Kulmizev and Nivre, 2023), hate speech detection (Ramponi and Tonelli, 2022), and visual question answering (Karamcheti et al., 2021). Furthermore, data maps have been shown to be an effective data selection tool for active learning (Zhang and Plank, 2021; Liu et al., 2022). The idea of using semantic simi- larity for zero-shot classification can traced back to Gabrilovich and Markovitch (2007). Chang et al. (2008) introduced the concept of dataless classi-
In this paper, have introduced the concept of Zero-Shot Data Maps, new method for under- standing annotated datasets. Our evaluation on various datasets demonstrates the effectiveness of Zero-Shot Data Maps in providing precise visual representation of labelled datasets.
Our Zero-Shot Data Maps match the pre- cision of maps produced with training dynamics, while providing the added benefit of being available instantly.
However, training dynamic maps can be drawn for a broader array of tasks, making zero-shot maps as a valuable complement,
rather than direct replacement. Furthermore. our approach proves to an effective tool for spot- ting potential annotation errors: the confidence of ensembled zero-shot models acts as a reliable indi- cator for identifying incorrect gold labels.
The quality of zero-shot maps largely depends on carefully crafted label descriptions and their en- semble combination. An intriguing direction for future research lies not only in improving the gen- cration these descriptions, potentially through automated methods, but also experimenting with

---

different zero-shot architectures approach more task types than bi-encoders allow. The combination of our method with active learning and automated error detection algorithms could pave the way for more efficient dataset curation and improvement processes. We believe that our approach to create data maps can help researchers and practitioners improve data quality and ultimately build better models.
the Fifth Joint Conference on Lexical and Computa- tional Semantics, pages 136-141, Berlin, Germany. Association for Computational Linguisties.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Ming- Ming-Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representa- tion: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume AAAI'08, page 830-835. AAAI Press.
Limitations
The main limitation of our work is that zero-shot data maps from bi-encoders can be compiled effec- tively only for the same type of tasks that can be solved by bi-encoder network architectures. While other, more flexible but resource-hungry, zero-shot methods can be adapted to build data maps, they cannot compete in terms of speed. Furthermore, the success of zero-shot data maps depends on the quality of label descriptions used in the zero-shot setup.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with v-usable information. In International Conference on Machine Learning.
Exgeniy Gabrilovich and Shaul Markovitch. 2007.
Computing semantic relatedness using wikipedia- based explicit semantic analysis. In International Joint Conference on Artificial Intelligence.
As a result, zero-shot data maps can also dis- play considerable variability across datasets, and the insights derived from them may not generalize effectively across different tasks.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms annotation tasks.
for text-
Dirk Groeneveld, Akshita Bhagia, and Pete Walsh. 2023. A12 Tango. https: //tt:///gtttttttt com/ allenai/ tango.
Acknowledgments
We thank the anonymous reviewers for their insightful comments and Thomas Mueller for his valuable feedback during the initial stages of this work. The work of Paolo Rosso was in the framework of the FairTransNLP re- funded
Antonio Gulli. 2005. AG's corpus of news articles.
http://groups.di..o wtt../d...iii..tt.ulll..... AG oeg_orppss.ss_nev__atiiilee.ss
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107-112. 12. New Orleans, Louisiana. Association for Computa- tional Linguistics.
search project (PID2021-1243610BB by MCIN/AEI/10. 13039/501 1033 and by ERDF, EU A way of making Europe. The work from Symanto has been partially funded by the Pro"Haters Proactive Profiling of Hate Speech Spreaders (CDTi IDI-20210776), the XAI- DisInfodemics: eXplainable AI for disinforma- tion and conspiracy detection during infodemics (MICIN PLEC2021-007681), the OBULEX- OB- servatorio del Uso de Lenguage sEXista en la red (IVACE IMINOD/2022/106) and the ANDHI ANomalous Diffusion of Harmful Information (CPP2021-0089994) R&D grants.
Xingwei He,, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023. Annollm: Making large language models to be better crowdsourced annotators.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. 2021. Mind your outliers! in- vestigating the negative impact of outliers on active learning for visual question answering. In Proceed- ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 7265-7281, Online. Association for Computational Linguistics.
References
Azad Abad and Alessandro Moschitti. 2016. Taking the best from the crowd:learning question passage classification from noisy data. In Proceedings of

---

Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023, Annotation Error Detection: An- alyzing the Past and Present for More Coherent Future. Computational Linguistics, 49(1):157-198.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putatiooal Linguistics, pages 2014-2037, Dubrovnik, Croatia. Association for Computational Linguistics.
Artur Kulmizev and Joakim Nivre. 2023. Investigat- ing UD treebanks via dataset difficulty measures. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 1076-1089, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
Thomas Müller, Guillermo Pérez-Toree and Mare Franco-Salvador. 2022. Few-shot learning with Siamese networks and label tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 8532-8545, Dublin, Ireland. Association for Computational Linguistics.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame. Julien Plu. Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Samh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac. Thibault Goehringer. Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pre- trained text-to-text models. In Findings of the As- sociation for Computational Linguisticx: ACI. 2022, pages 1864 1874, Dublin, Ireland. Association for Computational Linguistics.
Curtis Northcutt, Anis Athalye, and Jonas Mueleer. 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1.
OpenAi. 2022. Introducing chatgpt.
OpenAL. 2023. Gpt-4 technical report.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and Al collabora- tion for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826-6847, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily L. Denton, and A. Hanna. 2020. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Parterns, 2.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Seikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 2830.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 201 I. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 152-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140):1-67.
Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, and Dan Roth. 2022. Label semantic aware pre- pre-training for few- shot text classification. Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8318- 8334, Dublin, Ireland. Association for Computational Linguistics.
Alan Ramponi and Sara Tonelli. 2022. Features or spu- rious artifacts? data-centric baselines for fair and robust hate speech detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3027-3040, Seat- tle. United States. Association for Computational Linguistics.
Proceedings of the 60th

---

Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJ)) (EMNLP-IJCNLP),, pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush, 2020. Trans- formers: State-of -hee----thhhheeee art natural language processing. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Association for Computational Linguistics.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M. Rush. 2020. Learning from others' mistakes: Avoiding dataset biases without modeling them. ArXiv, 1.01300.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLI)) pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and Proceedings of the
natural language inference. 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
Mike Zhang and Barbara Plank. 2021. Cartography ac- tive learning. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 395- 406, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423.
Taylor Shin, Yasaman Razeghi, Robert I., Logan IV, Fric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222 4235, Online. Association for Computational Linguistics.
Xiang Zhang. Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28. Curran Associates, Inc.
A
Appendix
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast- but is it good? evaluating non-expert annotations for natural lan- guage tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Process- ing, pages 254-263, Honolulu, Hawaii. Association for Computational Linguistics.
A.1 Model Calibration
The effectiveness of zero-shot maps is contingent on the proper calibration of the zero-shot model. In Figure 5, we demonstrate the impact of calibra- tion, showcasing two maps, one compiled with a calibrated model and the other without. While both maps display a bell-shaped curvature, the curve's ss prominence is notably enhanced with calibration. As seen from Figure 5, the calibration step is crucial for compiling zero-shot maps that closely resemble those derived using training dynamics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington. USA. Association for Computational Linguistics.
For the calibration process, we fit an isotonic regressor, which is designed map our model's initial predicted logits to new, calibrated values.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi. Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online. Association for Computa- tional Linguistics.
The function used in this process is strictly mono- tone, which means that it retains the original order of predictions. Consequently, while this calibration process improves the visual representation of our datasct by providing more accurate probability timates, it does not affect the data selection part of our methodology, since the predicted class la- bels remain unchanged. Therefore, this calibration process improves the interpretability of our zero- shot data maps without altering their underlying
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 641-

---

054
0.3
correctness
0.2
confidence
0.50
0.50
025
0.48
0.01
0.02
0.2 variability
variability
Figure Zero-shot Data Maps SST2 compiled using sentence-15-large with calibration (right) and without (left).
structure. We note that calibration done exactly once per encoder and the calibrated model then applied to all tasks.
A.3 Item-Level Metries Comparison in Zero-Shot and Training Models
AG_NEWS
COLA
IMDB
SST2
A.2 Analysis of Variability in Label Descriptions
confidence
61.48
34.34
28.17
77.47
correctness
47.34
42.3
32.95
81.93
variability 27.82
22.8
31.88
36.34
To evaluate the stability of our maps concerning different sets of label descriptions, we introduce dropout maps. Starting with the complete coordi- nate matrix, these maps are created by randomly omitting pairs of label description combinations. We construct dropout maps with varying numbers of retained label description combinations (3, 5, 8. 13, and 21), omitting the rest. A Pearson cor- relation analysis is then conducted between the variability vectors of the dropout and full maps. Our findings in Table 6 indicate strong robustness, even when only combinations are retained. This confirms that the observed variability in the maps largely stems from label descriptions rather than input instances.
Table 7: Percentage of items categorized into the same bin (high, medium, low) for both training dynamics and zero-shot maps across four datasets. For example, in the SST2 dataset, 77.4 77.47% of the items were assigned to the same confidence bin both methods.
To assess the overlap between training dynamics (TD) and zero-shot (ZS) maps in terms of item cat- egorization, we binned the coordinates into three buckets: high, medium, and low. We then com- pared the percentage of instances that fell into the same bin in both methods. The results in Table 7 re- veal strong alignment between the TD and ZS ap- proaches. We note that we have conducted here the strictest possible test, measuring the exact matches between the two methods across the whole corpus for each dataset.
n
AG
NEWS
COLA
IMDB
SST2
3 0.60
0.58
0.58
0.65
A.4 Implementation Details
5
0.72
0.70
0.65
0.76
7
0.79
0.79
0.81
0.82
We use the Sentence Transformers (Reimers and Gurevych, 2019) and HuggingFace Transformers (Wolf et al., 2020) libraries for using the Trans- former models: specifically, for all our experiments sentence-15-large (Ni al., 2022), a 335 million parameter sentence-encoder based T5 (Raffel et al., 2020). Based on the MTEB bench- mark (Muennighoff et al., 2023), sentence-t5-larg. is the best- bert-peerorming model that we can run on
10
0.86
0.86
0.84
0.88
15
0.90
0.89
0.87
0.92
20
0.93
0.93
0.92
0.93
Table 6: Pearson correlation coefficients between origi- nal full map and dropout maps with varying number of ensemble members.

---

our GPU and the third best model overall for classi- fication tasks. The Al2-Tango toolkit (Groeneveld et al., 2023) handles our experiment management. We employ the AdamW optimizer in training dy- namics experiments and scikit-learn (Pedregosa et al., 1) for the experiments with frozen em- beddings. All the datasets for the experiments are sourced and loaded using the Huggingface Datasets library (Lhoest et al., 2021). )). All computations performed on single RTX A6000 GPU, with consistent batch size of 8 across all datasets, which is the maximum batch size that can be used to finetune sentence-t5-large on our GPU before en- countering out-of out-of-memory errors.
A.5 Dropping Ensemble Members
In the original work involving training dynamics, epoch burn-out scheme is employed to discard the initial steps until the training process stabilizes: by discarding the predictions from the initial train- ing epochs, this scheme prevents premature influ- ence from the initial, unstable steps of the training.
In our approach to creating zero-shot data maps, we replicate this burn-out effect by dropping en- semble members. This process mirrors the ef- fect of epoch burn-in from the training dynamics method. Specifically, selectively discard predic- tions from some ensemble members.
This method ensures that our metrics of confi- dence and variability are based on more reliable model predictions, thereby enhancing the robust- ness and reliability of the zero-shot data maps.
A.6 Label Descriptions
In Table 8, we present an overview of the datasets, labels, corresponding descriptions, and the associ- ated patterns used in our experiments. The label descriptions and patterns were manually crafted, drawing inspiration from previous studies zero- and few-shot learning (Schick and Schütze, 2021; Müller et al., 2022). Specifically, we used tem- plate with class names (e.g., 'Category: business, science, sports") " to match realistic scenario where here users would prefer to minimize time spent on crafting descriptions. The label descriptions used in our experiments were generated combinatorially based on the data presented in Table 8.

---

Dataset
Label
Deseriptions
Patlerns
Business
business, markets, money
Sci/Toch
science and tochnology, technology, science sports, fitness, races world, global, interaaional
Ag News
Ii is news. Category: D), Section
Sports
World
acceptable
acceptable, good, right, perfect
11, it (b., This sentence is grammatically [I-
Cola
unacceplahle
unacceplahle, not accepiable, wrong, incorrect
This semience is ()., The granmuticality of this ventence {).
IMDB
nog
nogative, bad, terrible, poor positive, great, awcsome, fantastic
This feels ()., Just (H. All all, movic was (}-
pos
negutive positive
negutive, bad, terrible, poor positive, great, awesome, fantastic
SST2
This feels ()., Just (H. All all, was (}-
negative, bad, terrible, poor
yelp polarity
[1, This feels []., Just (11, all, the restaurant was I.
positive, great, awesome, fantastic
Table 8: Overview of the labels, label descriptions and patterns used for each dataset. The placeholder () in the patterns host label descriptions. the Label column we include class names as they are encoded in the datasets accessed through the Huggingface Datasets (Lhoest et al., 2021) library.


Reviews:
[{"review_id": "eeF1bKFq3n", "reviewer": "Reviewer_CCvW", "summary": "", "questions": "In addition to the points raised in the \"Reasons to Reject\" section, I would like the authors to address the following queries:\n\nD. **Interpretation of Table 3:** In lines 383-385, you state: \"The results in Table 3 show that both methods yield comparable outcomes when used for data selection.\" However, for some categories, the difference between the two columns appears quite substantial. For instance, consider the 'high confidence' category for the CoLA dataset. Could you please elaborate on this?\n\nE. **Comparison of Zero-Shot Maps to Training Dynamics:** A detailed comparison between individual instances in Zero-Shot (ZS) and Training Dynamics (TD) maps would have been insightful. For example, it would be interesting to know what percentage of instances labeled as 'high-confidence' in ZS maps also fall under the same category in TD maps. Have you conducted some similar study? If so, incorporating these findings into the paper would be beneficial.\n\nF. **Computational Complexity with Label Descriptions:** In lines 216-217, you claim that the proposed method \"delivers efficient inference time regardless of the number of labels\". Is this efficiency due to the fact that label descriptions need to be encoded only once for the entire dataset? However, depending on the number of label descriptions required per class, encoding these descriptions might also be computationally expensive. Further few-shot datasets may have small ratio of input-instances/number of labels. Could you include a more detailed analysis of the computational complexity, considering both the number of labels and label descriptions as variables?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.\n\n**A. Label Descriptions**\n\nIn appendix A.4, we outline our method for creating label descriptions based on previous work. We used a template with class names (e.g., 'Category: {business, science, sports}') because it seems practical, considering users might want to spend less time on creating descriptions. All the label descriptions we used in our experiments are generated combinatorially from the data listed in Table 6.\nIn Section 2.2 of our paper, we mentioned that even if a label description seems well crafted, it might not always give the best results. We have shown the performance of individual label descriptions in Table 1 and Figure 2.\nTo test our approach further, we tried a different way of creating label descriptions. We used ChatGPT (3-5 Turbo, consulted on August 26, 2023) to generate new descriptions for the CoLA dataset. After manually checking and testing these descriptions, the results were similar to our original ones. We plan to add this data in our paper's next version.\n\n**B. Variability in Label Descriptions**\n\nTo address this point, we have conducted new experiments to better understand the effect of label description variability on our map's structure.\n\n- Data Processing: For each dataset we have been working with, the coordinate matrix was derived. This matrix offers a 2D representation, the columns of which indicate 'confidence', 'variability', and 'correctness' and the rows are individual instances. This is the data map that we use for plotting the images in the paper.\n- Introduction of Dropout Maps: 1000 dropout maps were constructed by randomly keeping various numbers of combinations of label descriptions (3, 5, 8, 13, and 21): e.g., in the table below, n=3 indicates that we build a new map with only 3 combinations of label descriptions. We note that for some datasets we have more than 700 combinations.\n- Correlation Assessment: We then ran a correlation analysis (Pearson) with the original variability vector from the full map against the variability vector from every dropout map.\nOur analysis reveals a strong correlation between the original full map and these dropout maps, as will be detailed in the table below. This correlation underscores the robustness of our approach even in the face of potential variability due to label descriptions. We note that already in the extreme case where we build a map with only 3 combinations, there is a good to strong correlation with the full map.\n\n|   n |   ag_news |   cola |   imdb |   sst2 |\n|----:|----------:|-------:|-------:|-------:|\n|   3 |      0.6  |   0.58 |   0.58 |   0.65 |\n|   5 |      0.72 |   0.7  |   0.65 |   0.76 |\n|   7 |      0.79 |   0.79 |   0.81 |   0.82 |\n|  10 |      0.86 |   0.86 |   0.84 |   0.88 |\n|  15 |      0.9  |   0.89 |   0.87 |   0.92 |\n|  20 |      0.93 |   0.93 |   0.92 |   0.93 |\n\n\n**C. Error Detection**\n\nGiven that large annotated corpora are usually not re-annotated or checked for errors independently, they are hard to find. We therefore report here an additional experiment using synthetic data. We randomly perturb 5% of the labels of the sst2 dataset and report the average precision in addition to the existing results obtained on the semi-automatically reviewed version of IMDB. Zero-shot maps outperform training dynamic maps by a large margin.\n\n\n|                  | imdb | sst2 |\n|------------------|------|------|\n| training dynamics| 30.2 | 49.0 |\n| zero shot        | 34.7 | 57.0 |\n\n\n**D. Interpretation of Table 3**\n\nWe hypothesize that the differences in performance, especially in the high-confidence category, might be rooted in the inherent challenges of the CoLA corpus. Specifically, a dual-encoder embedder might not possess adequate knowledge to accurately understand sentence grammaticality: this hypothesis is supported by the low performance of the classifier in this dataset.This potential limitation of the dual-encoder could be contributing to the observed discrepancies between the two methods for the CoLA dataset.\nWe will rewrite the sentence and make sure to frame this hypothesis appropriately in our paper for clearer interpretation. As we mentioned in the limitations section, not all tasks can be properly modeled by this framework and zero-shot maps are an efficient complement to training dynamics maps, more than a replacement. \n\n\n**E. Comparison of Zero-Shot Maps to Training Dynamics**\n\nTo address this question we have binned the coordinates in 3 bins (intuitively: high, low, medium) and compared the percentage of instances that fall in the same bin. We observe good to very strong agreement between the two methods. We note that we have conducted here the strictest possible test, measuring the exact matches between the two methods across the whole dataset for each corpus. \n\n| feature     |   ag_news |   cola |   imdb |   sst2 |\n|:------------|----------:|-------:|-------:|-------:|\n| confidence  |     61.48 |  34.34 |  28.17 |  77.47 |\n| correctness |     47.34 |  42.3  |  32.95 |  81.93 |\n| variability |     27.82 |  22.8  |  31.88 |  36.34 |\n\n\n\n\n\n**F. Computational Complexity with Label Descriptions**\n\nZero-shot dual-encoders are efficient at inference time because a single pass is required over the input data and the label descriptions, independently of the number of labels. Given that data maps are most useful for analyzing large annotated datasets, the number of instances is always going to exceed the number of labels; furthermore, label descriptions are usually much shorter than input texts. However, I have computed the required time (seconds) to encode different datasets using Colab (Nvidia T4 GPU) with sentence-t5-large. We have simulated a worst case scenario: for both texts and labels we have sampled random texts from the IMDB corpus, meaning that label descriptions are here also quite long.\n\n|   n_instances/n_labels |     3 |     5 |     7 |    13 |    50 |   100 |   1000 |\n|--------------:|------:|------:|------:|------:|------:|------:|-------:|\n|             3 |  0.24 |  0.31 |  0.37 |  0.59 |  1.85 |  3.45 |  32.01 |\n|             5 |  0.32 |  0.37 |  0.44 |  0.61 |  1.93 |  3.53 |  32.01 |\n|             7 |  0.37 |  0.45 |  0.5  |  0.71 |  1.98 |  3.62 |  32.22 |\n|            13 |  0.59 |  0.63 |  0.71 |  0.9  |  2.16 |  3.79 |  32.19 |\n|            50 |  1.7  |  1.76 |  1.87 |  2.18 |  3.32 |  4.91 |  33.57 |\n|           100 |  3.24 |  3.36 |  3.47 |  3.62 |  4.75 |  6.41 |  34.96 |\n|          1000 | 32.04 | 32.21 | 32.5  | 32.63 | 33.81 | 35.46 |  69.99 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a novel concept of Zero-Shot Data Maps, offering a more efficient approach to analyzing large datasets compared to traditional methods that rely on training dynamics. This contribution is significant, as it enables more expedient evaluation of extensive datasets. The authors substantiate the effectiveness of Zero-Shot Data Maps through qualitative analysis and downstream tasks, demonstrating their similarity to maps generated through training dynamics. Consequently, these Zero-Shot Data Maps can serve as a viable alternative for tasks such as error detection.", "reasons_to_reject": "A. **Label Descriptions:** The paper lacks some clarity on the process of creating label descriptions used in the zero-shot setup. Specifically, it does not detail the strategies employed to craft these descriptions or assess their quality. Does Table 6 contains all the descriptions? A comprehensive explanation of these aspects would enhance the understanding of the method's effectiveness and its impact on the resulting data maps.\n\nB. **Variability in Label Descriptions:** The paper does not sufficiently address the potential impact of variability in label descriptions on the generated data maps. Specifically, it is unclear how the authors ensure that the observed variability in data maps is attributed to the descriptions themselves rather than the input instances. For instance, high variability could potentially be a result of poorly crafted descriptions. An analysis to disentangle these effects would strengthen the paper's claims and ensure its generalizability to larger datasets.\n\nC. **Error Detection:** The evaluation of the automatic error detection capability of the proposed method is limited to a single dataset (IMDB). Given the significance of demonstrating that the error detection performance is on par with methods using Training Dynamics, it would be beneficial to include evaluations on additional datasets. This would provide a more robust validation of the method's effectiveness in error detection.", "questions_to_authors": "In addition to the points raised in the \"Reasons to Reject\" section, I would like the authors to address the following queries:\n\nD. **Interpretation of Table 3:** In lines 383-385, you state: \"The results in Table 3 show that both methods yield comparable outcomes when used for data selection.\" However, for some categories, the difference between the two columns appears quite substantial. For instance, consider the 'high confidence' category for the CoLA dataset. Could you please elaborate on this?\n\nE. **Comparison of Zero-Shot Maps to Training Dynamics:** A detailed comparison between individual instances in Zero-Shot (ZS) and Training Dynamics (TD) maps would have been insightful. For example, it would be interesting to know what percentage of instances labeled as 'high-confidence' in ZS maps also fall under the same category in TD maps. Have you conducted some similar study? If so, incorporating these findings into the paper would be beneficial.\n\nF. **Computational Complexity with Label Descriptions:** In lines 216-217, you claim that the proposed method \"delivers efficient inference time regardless of the number of labels\". Is this efficiency due to the fact that label descriptions need to be encoded only once for the entire dataset? However, depending on the number of label descriptions required per class, encoding these descriptions might also be computationally expensive. Further few-shot datasets may have small ratio of input-instances/number of labels. Could you include a more detailed analysis of the computational complexity, considering both the number of labels and label descriptions as variables?", "paper_topic_and_main_contributions": "This paper introduces Zero-Shot Data Maps, a  method for visualizing and interpreting labeled datasets without the need for model training. These 2D Data Maps [1] helps in interpreting dataset, by finding different types of instances(such as easy-to-learn or annotation errors). The method utilizes a single bi-encoder model with varying label descriptions to estimate the confidence and variability for each data point. The authors demonstrate that these Zero-Shot Data Maps can approximate Training Dynamics maps, particularly in tasks such as error detection, while being significantly faster to compute. In summary, the key contributions of this work are the introduction of an efficient zero-shot approach to data cartography, validation of its effectiveness as an alternative to training-based maps.\n\n**References**\n1. Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N.A., & Choi, Y. (2020). Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. Conference on Empirical Methods in Natural Language Processing.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m7atYf4RGI", "reviewer": "Reviewer_FBi8", "summary": "", "questions": "1. I am curious about using larger and stronger classifier.\n2. Could you provide a more detailed error analysis?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.\n\nRegarding a larger and stronger classifier: to build our classifier we use sentence-t5-large, which is already a SOTA encoder model in its class. We have also conducted several experiments with weaker encoders and, as long as the model is calibrated, results are in line with what we would expect to see.\n\nRegarding simplicity, one of the main goals of the paper was indeed from the beginning to reproduce data maps with a simpler system. I would argue that it is a strong point of the paper, agreeing with the reviewer, who also mentioned it as a reason for accepting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is easy to follow.\nProposed method is simple but effective.", "reasons_to_reject": "Overall, the model proposed by the author is very simple and straight forward, which might limit the technical contributions of the model.", "questions_to_authors": "1. I am curious about using larger and stronger classifier.\n2. Could you provide a more detailed error analysis?", "paper_topic_and_main_contributions": " Authors introduce a novel method for compiling data maps up to 14x faster. They compare their method against training dynamics-based maps and show that zero-shot maps can be used for automatic annotation error detection.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XCusaBK5NQ", "reviewer": "Reviewer_t41N", "summary": "", "questions": "1. How is the isotonic regressor trained for caliration? Does this procedure require some sort of ground-truth labels?\n\n2. How is the final data map sensitive to the label description phrasing? Section 2.2 does not dispel my concerns.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.\n\n**On Calibration.**\n\nWe want to emphasize that calibration is performed once for each encoder, not for every dataset/task. Specifically, we calibrate the zero-shot classifier using sentence-t5-large and apply it across all experiments. In practice, our goal is to provide users with a pre-calibrated zero-shot classifier: we believe that this configuration adds no subjective factors or negative effects on confidence.\n\nHere is the calibration process in details:\n\n- Initial Predictions. We start by obtaining predictions from the uncalibrated zero-shot classifier on a collection of labeled datasets.\n- Optimal Probability Estimation. For every instance, an optimal probability is estimated. Instances are organized based on their initial softmax probability. In this way, a neighborhood of 100 instances surrounding the current one is identified. The optimal probability for an instance is then calculated using the accuracy of this neighborhood, which relies on the ground-truth labels.\n- Logits Binning. All prediction logits are split uniformly to create bins. Logits are assigned to these bins using binary search.\n- Function Formulation and Weight Estimation. The calibration adjusts the initial logits using a monotonic function: $y=f(x)=x \\times wi(x)$​. Here, $x$ represents the initial logit and $y$ is the updated logit. The function $i(x)$ assigns a specific bin to $x$, while $wi​$ is a weight for bin $i$. These weights, also termed as scale factors (SF), are estimated to minimize the difference between the ideal probability and the softmax probability post $f(x)$ application.\n\nWill will upgrade the appendix to include all these details.\n\n**On Map sensitivity to label description phrasing.**\nI believe that this question is the same question asked by reviewer  CCvW. I report here the results from the experiments we conducted to address this point.\n\nWe have conducted new experiments to better understand the effect of label description variability on our map's structure.\n- Data Processing: For each dataset we have been working with, the coordinate matrix was derived. This matrix offers a 2D representation, the columns of which indicate 'confidence', 'variability', and 'correctness' and the rows are individual instances. This is the data map that we use for plotting the images in the paper.\n- Introduction of Dropout Maps: 1000 dropout maps were constructed by randomly keeping various numbers of combinations of label descriptions (3, 5, 8, 13, and 21): e.g., in the table below, n=3 indicates that we build a new map with only 3 combinations of label descriptions. We note that for some datasets we have more than 700 combinations.\n- Correlation Assessment: We then ran a correlation analysis (Pearson) with the original variability vector from the full map against the variability vector from every dropout map.\nOur analysis reveals a strong correlation between the original full map and these dropout maps, as will be detailed in the table below. This correlation underscores the robustness of our approach even in the face of potential variability due to label descriptions. We note that already in the extreme case where we build a map with only 3 combinations, there is a good to strong correlation with the full map.\n\n|   n |   ag_news |   cola |   imdb |   sst2 |\n|----:|----------:|-------:|-------:|-------:|\n|   3 |      0.6  |   0.58 |   0.58 |   0.65 |\n|   5 |      0.72 |   0.7  |   0.65 |   0.76 |\n|   7 |      0.79 |   0.79 |   0.81 |   0.82 |\n|  10 |      0.86 |   0.86 |   0.84 |   0.88 |\n|  15 |      0.9  |   0.89 |   0.87 |   0.92 |\n|  20 |      0.93 |   0.93 |   0.92 |   0.93 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The framework proposed in this paper is relatively simple.\n\n2. The method seems to be practical, at least based on the reported results.", "reasons_to_reject": "1. The method proposed by Swayamdipta et al. (2020) is based on training dynamics, which provides a certain level of theoretical interpretability (variability captures the model's uncertainty). While the zero-shot data map involves too many subjective factors, like the design of label description phrasing and the calibration. This may potentially have a negative impact on the confidence of applying this method in practical applications.\n\n2. The authors claim that the calibration function is strictly monotone and thus does not alter their underlying structure in Appendix A.1. However, the effect of calibration may have a significant impact when we need quantitative analysis or comparison between different categories.", "questions_to_authors": "1. How is the isotonic regressor trained for caliration? Does this procedure require some sort of ground-truth labels?\n\n2. How is the final data map sensitive to the label description phrasing? Section 2.2 does not dispel my concerns.", "paper_topic_and_main_contributions": "This paper proposes a method for building zero-shot data maps. Data Maps is useful for diagnosing large annotated datasets. It visualize the data points according to the confidence (the average probability assigned by the ensemble members to a specitfic label) and variability / uncertainty (the standard deviation of the model’s probabilities across the ensemble members) scores. In previous work (Swayamdipta et al., 2020), confidence and variability are calculated using training dynamics, which requires the fitting of a strong model to the dataset. This paper proposes to compute confidence on the true label and variability over the members of an ensemble of zero-shot models constructed with different — but semantically equivalent — label descriptions, i.e., textual representations of each class in a given label space, based on sentence-t5-large. Emprical results show that the zero-shot data maps (after calibration) generally match those produced by the traditional method while delivering up to a 14x speedup.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Figure 2 is cited before Figure 1 (Line 210 vs 255).\n\n2. Figure 2 presents 4 boxes, while in the caption it says 3 datasets. Additionally, the performance of state of the art and purely random could also be presented in the figure."}}, {"review_id": "xfyAAR7lwi", "reviewer": "Reviewer_b4Jj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. \n\nWhile we like the idea of adapting our methods to work with LLMs and have played with it, the final system would considerably compromise the efficiency we aim for with our current method: to get N different responses on a single data point, we would need to prompt the LLM N times with N different prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Given the recent trend of zero/few-shot methods in NLP, the problem of interest is interesting and timely. \n2. The method is well-motivated and simple, and the results are impressive. This efficiency enhancement makes the technique more practical and scalable for large datasets.\n3. Evaluation is comprehensive, covering visual comparison and downstream applications (e.g., data selection and error detection). ", "reasons_to_reject": "1. It would be more interesting to see if LLMs can be applied in this setting and if they make the story different, even on a small scale. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces Zero-Shot Data Maps, a novel approach for diagnosing large annotated datasets without any model training. The key idea is to replace model training, which is time-consuming, with a strong and efficient zero-shot model. Like Data maps, the confidence and variability over the true labels are computed. The paper then presents a comparative analysis between training dynamics and the proposed approach on various datasets. The findings show that Zero-Shot Data Maps match Data Maps while achieving up to a 14x speedup. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents Zero-Shot Data Maps based on a single bi-encoder model with diverse label descriptions to assess the confidence and variability of each data point. The authors illustrate that Training Dynamics maps can be approximated using  Zero-Shot Data Maps while offering significantly faster computation. 


Reviewers are divided on this submission. I think the review by Reviewer FBi8 has a low quality and needs to be discarded. While the remaining two reviews have acceptable qualities, the two reviews with positive opinions are more extensive and the mentioned strengths outweigh the weaknesses raised by the negative review. I have read the reviews and the follow-up discussions and I think it is reasonable to accept this paper into the Findings track.
Decision: Accept-Findings