Venue: ACL 2023

# Analyzing Text Representations by Measuring Task Alignment

Cesar Gonzalez-Gutierrez, Audi Primadhanty, Francesco Cazzaro, Ariadna Quattoni

Universitat Politecnica de Catalunya, Barcelona, Spain {cesar.gonalezz.gutierrez, audi.primadhanty, francesco.cazzaro}@upc.edu, aquattoni@cs.upc.edu

###### Abstract

Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios. What makes a representation good for text classification? Is it due to the geometric properties of the space or because it is well aligned with the task? We hypothesize the second claim. To test it, we develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity. Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.

## 1 Introduction

Recent advances in text classification have shown that representations based on pre-trained language models are key, especially in few-shot learning scenarios (Ein-Dor et al., 2020; Lu et al., 2019). It is natural to ask: What makes a representation good for text classification in this setting? Is the representation good due to intrinsic geometric properties of the space or because it is well _aligned_ with the classification task? The goal of this paper is to answer this question to better understand the reason behind the performance gains obtained with pre-trained representations.

Our hypothesis is that representations better aligned with class labels will yield improved performance in few-shot learning scenarios. The intuition is simple: in this setting, the limited number of labeled samples will only provide a sparse coverage of the input domain. However, if the representation space is properly aligned with the class structure, even a small sample can be representative. To illustrate this, take any classification task. Suppose we perform clustering on a given representation space that results in a few pure clusters (with all samples belonging to the same class). Then, any training set that 'hits' all the clusters can be representative. Notice that there is a trade-off between the number of clusters and their purity. A well-aligned representation is one for which we can obtain a clustering with a small number of highly pure clusters. Based on this, we propose a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity: Task Hierarchical Alignment Score (Thas).

To test our hypothesis that task alignment is key we conduct experiments on several text classification datasets comparing different representations. Our results show that there is a clear correlation between the Thas of a representation and its classification performance under the few-shot learning scenario, validating our hypothesis and showing that task alignment can explain performance. In contrast, our empirical study shows that intrinsic geometric properties measured by classical clustering quality metrics fail to explain representation performance in the few-shot learning scenario.

Our study suggests an answer to our main question: A good efficient representation (i.e. one that enables few-shot learning) is a representation that induces a good alignment between latent input structure and class structure. Our main contributions are: 1) We develop a score based on hierarchical clustering (SS2) that measures the extent to which a representation space is aligned with a given class structure and 2) We conduct an empirical study using several textual classification datasets

Figure 1: Three-step process for computing Thas.



(SS3) that validates the hypothesis that the best representations are those with a latent input structure that is well aligned with the class structure.

## 2 Task Hierarchical Alignment Score

We now present the Task Hierarchical Alignment Score (Thas) designed to measure the alignment between a textual representation and the class label for a given task. The idea is quite simple, in a good representation space, points that are close to each other should have a higher probability of belonging to the same class. Therefore, we could perform clustering of the points and obtain _high purity_ clusters, where most points belong to the same class. We assume that we are given: a dataset \(S=\{(\mathbf{x}_{i},y_{i})\}^{n}_{i=1}\) of \(n\) labeled data points where \(\mathbf{x}\in\mathcal{X}\) is a text fragment and \(y\in\mathcal{Y}\) its corresponding class label (e.g., a sentiment classification label) and a representation function \(r:\mathcal{X}\rightarrow\mathbb{R}^{d}\) mapping points in \(\mathcal{X}\) to a \(d\)-dimensional representation space \(\mathbb{R}^{d}\) (e.g., a sparse bag-of-words).

Our goal is to compute a metric \(\tau(S,r)\) that takes some labeled domain data and a representation function and computes a real value score. Fig. 1 illustrates the steps involved in computing Thas. There are three main steps: 1) hierarchical clustering, 2) computing clustering partition alignments, and 3) computing the aggregate metric. In the first step, we compute the representation of each point and build a data dendrogram using hierarchical clustering. The data dendrogram is built by merging clusters, progressively unfolding the latent structure of the input space. Traversing the tree, for each level we get a partition of the training points into \(k\) clusters. In step 2, for each partition, we measure its alignment with the class label distribution producing an alignment curve as a function of \(k\). Finally, we report the area under this curve. Algorithm 1 summarizes the whole procedure. Implementation details and performance information can be found in A.1.

```
Input: Dataset \(S=\{(\mathbf{x}_{i},y_{i})\}^{n}_{i=1}\), representation function \(r\) Output:\(\tau(S,r)\)
1 Get representation: \(\mathbf{R}=\{r(\mathbf{x}_{i})~{}|~{}\mathbf{x}_{i}\in\mathbf{X}\}\) Run Hierarchical Clustering: \(\mathcal{D}=\text{HC}(\mathbf{R})=\{\mathcal{P}_{k}\}^{n}_{k=1}\) Traverse the dendrogram: foreachpartition \(\mathcal{P}_{k}\subset\mathcal{D}\)do
4 Predict scores for all points: foreachpoint\(\mathbf{x}_{i}\in\mathbf{X}\)in\(i=1,\ldots,n\)where\(r(\mathbf{x}_{i})\in C\subset\mathcal{P}_{k}\)do
5 Label prediction scores: foreach\(y^{\prime}_{j}\in\mathcal{Y}\)in\(j=1,\ldots,|\mathcal{Y}|\)do\(\hat{\mathbf{Y}}_{k,i,j}=s(\mathbf{x}_{i},y^{\prime}_{j})\) Partition alignment score: \(a(\mathcal{P}_{k})=\text{AUC}_{y^{+}}(\hat{\mathbf{Y}}_{k},\mathbf{Y})\) Final aggregate metric: \(\tau(S,r)=\frac{1}{n}\sum_{k=1}^{n}a(\mathcal{P}_{k})\)
```

**Algorithm 1**Thas

### Hierarchical Clustering

In the first step, we will consider the input points \(\mathbf{X}=\{\mathbf{x}_{i}~{}|~{}(\mathbf{x}_{i},y_{i})\in S\}\) and the representation function \(r\) to obtain a representation of all points \(\mathbf{R}=\{r(\mathbf{x}_{i})~{}|~{}\mathbf{x}_{i}\in\mathbf{X}\}\).

We then apply Hierarchical Clustering (HC) to the points in \(\mathbf{R}\) obtaining a dendrogram \(\mathcal{D}=\text{HC}(\mathbf{R})=\{\mathcal{P}_{k}\}^{n}_{k=1}\) that defines a set of \(n\) cluster partitions. Fig. 1 (left) shows a diagram of a dendrogram. The root of this tree is the whole set and, at the leaves, each point corresponds to a singleton. At intermediate levels, top-down branching represents set splitting.

For each level \(k=1,\ldots,n\) of the dendrogram there is an associated clustering partition of the input points into \(k\) clusters \(\mathcal{P}_{k}=\{C_{j}\}^{k}_{j=1}\). That is, for any particular level we have a family of \(k\) non-empty disjoint clusters that cover the representation \(\mathbf{R}=\bigcup_{j=1}^{k}C_{j}\), where each representation point \(r(\mathbf{x})\in\mathbf{R}\) is assigned to one of the \(k\) clusters.

### Partition Alignment Score

We use the gold labels \(\mathbf{Y}=\{y_{i}~{}|~{}(\mathbf{x}_{i},y_{i})\in S\}\) to compute an alignment score \(a(\mathcal{P}_{k})\) for each partition \(\mathcal{P}_{k}\subset\mathcal{D}\). We compute it in two parts.

First, for every point \(\mathbf{x}\in\mathbf{X}\) and label \(y^{\prime}\in\mathcal{Y}\) we compute a label probability score by looking at the gold label distribution of the cluster \(C\) to which the point belongs in the clustering partition:

\[s(\mathbf{x},y^{\prime})=\frac{1}{|C|}\#[y^{\prime}\in C] \tag{1}\]

where \(\#[y^{\prime}\in C]\) is the number of samples in cluster \(C\) with gold label \(y^{\prime}\). Intuitively, this assigns to a point \(\mathbf{x}\) a label probability that is proportional to the distribution of that label in the cluster \(C\).

Second, we use the label probability scores of all points \(\hat{\mathbf{Y}}_{k}=\{s(\mathbf{x}_{i},y^{\prime}_{j})~{}|~{}\mathbf{x}_{i}\in\mathbf{X},y^{ \prime}_{j}\in\mathcal{Y}\}\) and the 

dataset gold labels \({\mathbf{Y}}\) to compute a partition alignment score. We choose as a single metric the area under the precision-recall curve (AUC) because it has the nice property that it applies to tasks with both balanced and unbalanced class distributions.1 More specifically, we compute the AUC of the target (positive) class \(y^{+}\in\mathcal{Y}\) of the dataset (more details in the experimental part in SS3):

Footnote 1: F1 could be a valid alternative, but this metric requires the validation of decision thresholds.

\[a(\mathcal{P}_{k})=\text{AUC}_{y^{+}}(\hat{\mathbf{Y}}_{k},{\mathbf{Y}}) \tag{2}\]

### Final Aggregate Metric: Thas

Once we have an alignment score for every level of the hierarchical dendrogram, we are ready to define our final Task Hierarchical Alignment Score (Thas). Consider the alignment scoring function \(a\) applied to the partition corresponding to the lowest level of the dendrogram. The alignment score will be \(a(\mathcal{P}_{n})=1\) because every cluster in this partition is a singleton and therefore \(\#[y^{\prime}\in C]\) will be \(1\) for the gold label and \(0\) for any other label. At the other end, for the partition corresponding to the root of the dendrogram (where all points belong to a single cluster), the alignment score \(a(\mathcal{P}_{1})\) is the AUC corresponding to assigning to every point \({\mathbf{x}}\in{\mathbf{X}}\) a prediction score for each label \(y^{\prime}\in\mathcal{Y}\) equal to the relative frequency of \(y^{\prime}\) in \({\mathbf{Y}}\).

Consider now the alignment score as a function of the size of the partition. As we increase \(k\) we will get higher scores. A good representation is one that can get a high score while using as few clusters as possible. Instead of choosing a predefined level of granularity, we propose to leverage the alignment information across all levels. To achieve this, we consider the alignment score as a function of the number of clusters and measure the area under \(a(\mathcal{P}_{k})\).2 We are ready to define our final metric:

Footnote 2: We could consider weighting methods that neutralize uninformative areas in the curve. In particular, we could subtract the scores originating from a random clustering. However, this contribution is solely determined by the sample size and the prior distribution. As a result, it would not have any impact when comparing representations.

\[\tau(S,r)=\frac{1}{n}\sum_{k=1}^{n}a(\mathcal{P}_{k}) \tag{3}\]

## 3 Experimental Setup

In this section we empirically study the correlation of few-shot learning performance with 1) Thas and 2) an unsupervised clustering quality metric.

We use four text classification datasets with both balanced and imbalanced label distributions: IMDB (IM; Maas et al., 2011), WikiToxic (WT; Wulczyn et al., 2017), Sentiment140 (S1; Maas et al., 2011) and CivilComments (CC; Borkan et al., 2019).

We will compare the following representations: a sparse bags-of-words (BoW); BERT embeddings (Devlin et al., 2019) using two token average pooling strategies (BERTall and BERTcls); GloVe (Pennington et al., 2014); and fastText (Bojanowski et al., 2017; Joulin et al., 2016).

For further details, please refer to A.2.

### Few-Shot Performance vs. Thas

Since the focus of these experiments is comparing representations, we follow previous work on probing representations and use a simple model Tenney et al. (2019); Lu et al. (2019). More precisely, we use a linear max-entropy classifier trained with \(l2\) regularization.

To simulate a few-shot learning scenario, we create small training sets by selecting \(N\) random samples, from \(100\) to \(1000\) in increments of \(100\). For each point \(N\) in the learning curve we create an

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{5}{c}{**ALC**} & \multicolumn{5}{c}{**THas**} & \multicolumn{5}{c}{**ADBI**} \\ \cline{2-18}
**Repr.** & IM & WT & CC & S1 & \(\mu\) & IM & WT & CC & S1 & \(\mu\) & IM & WT & CC & S1 & \(\mu\) \\ \hline BERT\({}_{\text{all}}\) & _.84_ & _.50_ & _.32_ & _.79_ & **.61** & _.84_ & _.67_ & _.27_ & _.75_ & **.63** & 2.87 & 3.03 & 3.31 & 3.25 & 3.11 \\ GloVe & .80 & .48 & .26 & .74 & .57 & .80 & .63 & .26 & .73 & .60 & _2.62_ & _2.12_ & 2.01 & _2.47_ & **2.31** \\ BERT\({}_{\text{cls}}\) & .80 & .48 & .23 & .74 & .56 & .80 & .56 & .22 & .74 & .58 & 2.81 & 2.97 & 3.15 & 2.92 & 2.96 \\ fastText & .75 & .41 & .18 & .66 & .50 & .77 & .57 & .21 & .71 & .56 & 2.78 & 2.13 & _1.93_ & _2.47_ & 2.33 \\ BoW & .76 & .32 & .11 & .59 & .45 & .71 & .50 & .20 & .68 & .52 & 3.14 & 3.83 & 4.23 & 3.86 & 3.76 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Learning curve performance (ALC), task alignment (Thas), and unsupervised clustering quality (ADBI) for different representations and datasets. (Rows are sorted by average ALC.) 

80%/20% 5-fold cross-validation split to find the optimal hyper-parameters. We then train a model using the full \(N\) training samples and measure its performance on the test set. We repeat the experiment with 5 random seeds and report the mean results. As the evaluation metric, we use accuracy for the balanced datasets (IMDB and Sentiment140) and F1 for the imbalanced datasets (WikiToxic and CivilComments).

We generate learning curves for each dataset and representation (A.3). To study the correlation between task alignment and few-shot learning performance, it is useful to have a single score that summarizes the learning curve: We use the area under the learning curve (ALC). Representations with a larger ALC perform better in the few-shot learning scenario.3 We observe that BERTall is consistently the best representation followed by BERTcls and GloVe performing similarly. Representations based on word embeddings are better than the sparse baseline for all datasets, except for fastText which does not exhibit a consistent improvement.

Footnote 3: Alternatively, we could have picked a single point but we believe that ALC provides a more robust measure of few-shot learning performance and allows for a more concise analysis.

To test for correlation, we also computed Thas for each representation and dataset. (The corresponding curves can be found in A.3.) Since this metric is a measure of the alignment between a label distribution and an input representation, there is a Thas score per label.4 In the classification tasks that we consider there is always a single target class (e.g., toxicity for WikiToxic). We measure the alignment score with respect to this class.

Footnote 4: We could also aggregate the scores of different classes, for example taking the average of the scores over all labels.

Table 1 summarizes the results showing ALC (left) and corresponding Thas (center) for all representations and datasets. Overall, BERall is the best representation for few-shot learning followed by GloVe and BERTcls. All the representations based on pre-trained word embeddings significantly outperform the baseline sparse BoW representation. Thas predicts accurately the relative ranking between representations and the larger gap between BERall and the rest. Fig. 2 shows a scatter plot of Thas as a function of ALC (blue dots; each point corresponds to a dataset and representation). We compute the correlation coefficients, which are displayed in Table 2. We observe a clear positive correlation between the two metrics, providing supporting evidence for our main hypothesis that a good representation under few-shot learning is a representation that is well aligned with the classification task.

### Unsupervised Clustering Quality

We now look at standard metrics of cluster quality and test if they can explain few-shot learning performance. We use the Davies and Bouldin (1979) index (DBI) to measure the quality of the cluster partitions at every level of the dendrogram. This metric measures the compactness of each cluster and their separation, with better cluster partitions scoring lower. Similar to the computation of Thas described in SS2, we compute DBI as a function of the number of clusters \(k\) corresponding to each level of the dendrogram. As an aggregate metric, we calculate the area under these curves to obtain a single ADBI score. (The curves are shown in A.3.)

The right side of Table 1 shows the results for the same datasets and representations used for Thas. GloVe induces the best clusters according to the ADBI metric. BERTall does not produce particularly good clusters despite being the strongest few

\begin{table}
\begin{tabular}{l l l} \hline \hline (\(\mu\))ALC vs & \(r_{p}\) (p-value) & \(r_{s}\) (p-value) \\ \hline Thas & \(0.98\) (\(<10^{-12}\)) & \(0.99\) (\(<10^{-17}\)) \\ ADBI & \(0.11\) (\(0.62\)) & \(0.07\) (\(0.76\)) \\ \(\mu\)Thas & \(0.98\) (\(0.002\)) & \(1.0\) (\(0.017\)) \\ \(\mu\)ADBI & \(-0.41\) (\(0.48\)) & \(-0.3\) (\(0.68\)) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Pearson correlation coefficient (\(r_{p}\)) and Spearman’s correlation coefficient (\(r_{s}\)) with the corresponding p-values for ALC vs. Thas and ALC vs. ADBI, and similar analysis for mean scores across all datasets.

Figure 2: Few-shot performance (ALC) vs. task alignment (Thas) and clustering quality (ADBI).

 

shot representation. Fig. 2 (red crosses) and Table 2 show that there is a low correlation between the two metrics. This suggests that the geometric properties of the clusters alone can not explain few-shot performance.

## 4 Related Work

Representation choice has recently gained significant attention from the active learning (AL) community Schroder and Niekler (2020); Shnarch et al. (2022); Zhang et al. (2017). Some work has attempted to quantify what representation is best when training the initial model for AL, which is usually referred to as the cold start problem Lu et al. (2019). The importance of word embeddings has been also studied in the context of highly imbalanced data scenarios Sahan et al. (2021); Naseem et al. (2021); Hashimoto et al. (2016); Kholghi et al. (2016). Most research conducted by the AL community on textual representations has focused on determining _which_ representations lead to higher performance for a given task. However, our paper aims to investigate _why_ a certain representation performs better in the few-shot scenario.

Our work, focused on examining properties of various textual representations, is closely related to recent research on evaluating the general capabilities of word embeddings. Many studies are interested in testing the behavior of such models using probing tasks that signal different linguistic skills Conneau et al. (2018); Conneau and Kiela (2018); Marvin and Linzen (2018); Tenney et al. (2019); Miaschi and Dell'Orletta (2020). Others have targeted the capacity of word embeddings to transfer linguistic content Ravishankar et al. (2019); Conneau et al. (2020).

Looking at approaches that analyze the properties of representations directly, without intermediate probes, Saphra and Lopez (2019) developed a correlation method to compare representations during consecutive pre-training stages. Analyzing the geometric properties of contextual embeddings is also an active line of work Reif et al. (2019); Ethayarajh (2019); Hewitt and Manning (2019). While these previous works focus on analyzing representation properties independently, without considering a specific task, our study investigates the relationship between representations and task labels. We conduct a comparison between this relationship and the unsupervised analysis of representation properties.

Our work falls in line with broader research on the relationship between task and representation. Yauney and Mimno (2021) proposed a method to measure the alignment between documents and labels in a given representation space using a data complexity measure developed in the learning-theory community. In the computer vision area, Frosst et al. (2019) introduced a loss metric and investigated the entanglement of classes in the representation space during the learning process. Zhou and Srikumar (2021) proposed a heuristic to approximate the version space of classifiers using hierarchical clustering, highlighting how representations induce the separability of class labels, thereby simplifying the classification task. In contrast, our work specifically examines the few-shot performance and emphasizes the importance of unbalanced scenarios. We find that in these more realistic situations, the choice of representation plays a critical role, paving the way for advanced strategies in active learning.

## 5 Conclusion

In this paper, we asked the question: What underlying property characterizes a good representation in a few-shot learning setting? We hypothesized that good representations are those in which the structure of the input space is well aligned with the label distribution. We proposed a metric to measure such alignment: Thas. To test our hypothesis, we conducted experiments on several textual classification datasets, covering different classification tasks and label distributions (i.e. both balanced and unbalanced). We compared a range of word embedding representations as well as a baseline sparse representation.

Our results showed that when labeled data is scarce the best-performing representations are those where the input space is well aligned with the labels. Furthermore, we showed that the performance of a representation can not be explained by looking at classical measures of clustering quality.

The main insight provided in this work could be leveraged to design new strategies in active learning. The fact that good representations induce clusters of high purity at different granularities creates opportunities for wiser exploration of the representation space in an active manner. Similar to the work of Dasgupta and Hsu (2008), we could employ the data dendrogram to guide this exploration 



### Limitations

In this paper, we focused on analyzing the properties of textual representations in the few-shot learning scenario. Its applicability to broader annotation scenarios could be presumed but is not supported by our empirical results.

Our experimental setup is based on binary classification tasks using English datasets. While our approach is general and could be easily extended to multi-class scenarios, more work would be required to extend it to other more complex structured prediction settings such as sequence tagging.

We see several ways in which this work could be extended. The most obvious extension consists of trying to generalize the notion of alignment to other tasks beyond sequence classification, such as sequence tagging. In this paper, we have used Thas to understand the quality of a given textual representation. However, since Thas is a function of a labeling and a representation, it could also be used to measure the quality of a labeling Yan and Huang (2018), given a fixed representation. For example, this might be used in the context of hierarchical labeling, to measure which level of label granularity is better aligned with some input representation.

The goal of this paper was to provide an explanation for the success of pre-trained word embeddings for text classification in the few-shot learning scenario. We believe that with our proposed methodology we have successfully achieved this goal. However, it should be clear to the reader that we do not provide a method for picking the best representation, i.e. for model selection. This is because our analysis requires access to labeled data and if labeled data is available the best way to select a model will be via cross-validation.

## Acknowledgements

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme under grant agreement No 853459. The authors gratefully acknowledge the computer resources at ARTEMISA, funded by the European Union ERDF and Comunitat Valenciana as well as the technical support provided by the Instituto de Fisica Corpuscular, IFIC (CSIC-UV). This research is supported by a recognition 2021SGR-Cat (01266 LQMC) from AGAUR Generalitat de Catalunya.

## References

* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Technical Report arXiv:1607.04606, arXiv. ArXiv:1607.04606 [cs].
* Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification. _arXiv:1903.04561 [cs, stat]_. ArXiv: 1903.04561.
* Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Conneau et al. (2018) Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single S&!#* vector: Probing sentence embeddings for linguistic properties. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
* Conneau et al. (2020) Alexis Conneau, Shijie Wu, Haoram Li, Luke Zettelmoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6022-6034, Online. Association for Computational Linguistics.
* ICML '08_, pages 208-215, Helsinki, Finland. ACM Press.
* Davies and Bouldin (1979) David Davies and Don Bouldin. 1979. A Cluster Separation Measure. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, PAMI-1:224-227.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ein-Dor et al. (2020) Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shanarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active Learning for BERT: An Empirical Study. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7949-7962, Online. Association for Computational Linguistics

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 55-65, Hong Kong, China. Association for Computational Linguistics.
* Frosst et al. (2019) Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. 2019. Analyzing and improving representations with the soft nearest neighbor loss. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 2012-2020. PMLR.
* Harris et al. (2020) Charles R. Harris, K. Jarrod Millman, Stefan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernandez del Rio, Mark Wiebe, Pearu Peterson, Pierre Gerard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. 2020. Array programming with NumPy. _Nature_, 585(7825):357-362.
* Hashimoto et al. (2016) Kazuma Hashimoto, Georgios Kontonatsios, Makoto Miwa, and Sophia Ananiadou. 2016. Topic detection using paragraph vectors to support active learning in systematic reviews. _Journal of Biomedical Informatics_, 62:59-65.
* Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of Tricks for Efficient Text Classification. Technical Report arXiv:1607.01759, arXiv. ArXiv:1607.01759 [cs].
* Kholghi et al. (2016) Mahnoosh Kholghi, Lance De Vine, Laurianne Sitbon, Guido Zuccon, and Anthony Nguyen. 2016. The Benefits of Word Embeddings Features for Active Learning in Clinical Information Extraction. In _Proceedings of the Australasian Language Technology Association Workshop 2016_, pages 25-34, Melbourne, Australia.
* Lhoest et al. (2019) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clement Delangue, Theo Matsusiere, Lysandre Debut, Stas Bekanem, Pierric Cistac, Tibbault Goehringer, Victor Mustar, Francois Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Lu et al. (2019) Jinghui Lu, Maeye Henchion, and Brian Mac Namee. 2019. Investigating the Effectiveness of Representations Based on Word-Embeddings in Active Learning for Labelling Text Datasets. ArXiv:1910.03505 [cs, stat].
* Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
* Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
* Miaschi and Dell'Orletta (2020) Alessio Miaschi and Felice Dell'Orletta. 2020. Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation. In _Proceedings of the 5th Workshop on Representation Learning for NLP_, pages 110-119, Online. Association for Computational Linguistics.
* Murtagh (1983) F. Murtagh. 1983. A Survey of Recent Advances in Hierarchical Clustering Algorithms. _The Computer Journal_, 26(4):354-359.
* Naseem et al. (2021) Usman Naseem, Mattobb Khushi, Shah Khalid Khan, Kamran Shaukat, and Mohammad Ali Moni. 2021. A Comparative Analysis of Active Learning for Biomedical Text Mining. _Applied System Innovation_, 4(1):23.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoi Steiner, Luc Fang, Junje Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in 

Python. _Journal of Machine Learning Research_, 12:2825-2830.
* Pennington et al. (2014) Jeffrey Penington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Ravishankar et al. (2019) Vinit Ravishankar, Lilja Ovrelid, and Erik Velldal. 2019. Probing multilingual sentence representations with X-probe. In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP-2019)_, pages 156-168, Florence, Italy. Association for Computational Linguistics.
* Reif et al. (2019) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and Measuring the Geometry of BERT. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Sahan et al. (2021) Marko Sahan, Vaclav Smidl, and Radek Mariak. 2021. Active Learning for Text Classification and Fake News Detection. In _2021 International Symposium on Computer Science and Intelligent Controls (ISCSIC)_, pages 87-94. IEEE Computer Society.
* Saphra and Lopez (2019) Naomi Saphra and Adam Lopez. 2019. Understanding learning dynamics of language models with SVCCA. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers)_, pages 3257-3267, Minneapolis, Minnesota. Association for Computational Linguistics.
* Schroder and Niekler (2020) Christopher Schroder and Andreas Niekler. 2020. A Survey of Active Learning for Text Classification using Deep Neural Networks. ArXiv:2008.07267 [cs] version: 1.
* Shnarch et al. (2022) Eyal Shnarch, Ariel Gera, Alon Halfon, Lena Dankin, Leshem Choshen, Ranit Aharonov, and Noam Slomin. 2022. Cluster & tune: Boost cold start performance in text classification. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7639-7653, Dublin, Ireland. Association for Computational Linguistics.
* Tenney et al. (2019) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. ArXiv:1905.06316 [cs].
* Ward (1963) Joe H. Ward. 1963. Hierarchical Grouping to Optimize an Objective Function. _Journal of the American Statistical Association_, 58(301):236-244.
* Wolf et al. (2018) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wulczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In _Proceedings of the 26th International Conference on World Wide Web_, WWW '17, pages 1391-1399, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
* Yan and Huang (2018) Yi-Fan Yan and Sheng-Jun Huang. 2018. Cost-effective active learning for hierarchical multi-label classification. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_, pages 2962-2968. International Joint Conferences on Artificial Intelligence Organization.
* Yauney and Mimno (2021) Gregory Yauney and David Mimno. 2021. Comparing text representations: A theory-driven approach. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5527-5539, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Zhang et al. (2017) Ye Zhang, Matthew Lease, and Byron Wallace. 2017. Active Discriminative Text Representation Learning. _Proceedings of the AAAI Conference on Artificial Intelligence_, 31(1).
* Zhou and Srikumar (2021) Yichu Zhou and Viveek Srikumar. 2021. DirectProbe: Studying representations without classifiers. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5070-5083, Online. Association for Computational Linguistics.

 

Appendix

### Thas Implementation Details

The data dendrogram is obtained via hierarchical agglomerative clustering. More precisely, we use a bottom-up algorithm that starts with each sample as a singleton cluster and consecutively merges clusters according to a similarity metric and merge criterion until a single cluster is formed.

We apply Ward's (1963) method, which uses the squared Euclidean distance between samples and then minimizes the total within-cluster variance by finding consecutive pairs of clusters with a minimal increase. The clustering algorithm produces a list of merges that represent a dendrogram and can be traversed to generate a clustering partition for each value of \(k\). It was implemented using Scikit-learn Pedregosa et al. (2011) and NumPy Harris et al. (2020).

Expressed as a nearest-neighbor chain algorithm, Ward's method has a time complexity of \({\cal O}(n^{2})\)Murtagh (1983). Thas experiments have been performed using sub-samples of size \(10\)K and averaged over \(5\) seeds. Using \(32\) CPUs and \(16\)GiB of RAM, each agglomerative clustering took on average \(3.3\) minutes. Each task alignment curve took \(3\) minutes on average. In contrast, DBI curves took \(7.8\) hours on average.

### Experimental Details

**Datasets.** Table 3 shows the statistics of the datasets used in this paper. They were extracted from HuggingFace Datasets Lhoest et al. (2021). For WikiToxic and CivilComments, we have applied a pre-processing consisting of removing all markup code and non-alpha-numeric characters.

Representations.The following is a detailed description of the text representations used in our experiments:

* BoW: this is a standard sparse term frequency bag-of-words representation.
* BERTall: word embeddings from Devlin et al.'s Devlin et al. (2019) BERTBASE uncased model, average pooling of 2nd to last layers and average pooling of all tokens.
* BERTcls: the same as above but using the [CLS] token alone.
* GloVe: Pennington et al.'s Poennington et al.'s (2014) word vectors pre-trained on Common Crawl with average pooling.
* fastText: word vectors from Bojanowski et al. Bojanowski et al. (2017); Joulin et al. (2016) pre-trained on Wikipedia with average pooling.

BERT representations were extracted using the HuggingFace Transformers library Wolf et al. (2020) implemented in PyTorch Paszke et al. (2019).

Models.The parameters for max-entropy learning curves were validated using \(5\)-fold cross-validation and the results averaged over sub-samples from \(5\) seeds.

### Curves

Fig. 3 presents the curves used to compute the main results in SS3. The left column contains the learning curves used to compute the few-shot learning performance of the different datasets and representations. The center column shows task alignment scores as a function of the number of clusters. Thas is computed as the area under these curves. The pre-trained word embeddings, in particular BERT, tend to achieve the best results. In the curves, they show higher values of alignment for a small number of clusters. The relative performance of the representations in the learning curves is paralleled in the task hierarchical alignment curves. BERTall (i.e. using average pooling over all tokens) seems to be superior to BERTcls (i.e. using only the [CLS] token).

The right column in Fig. 3 shows the DBI curves as a function of the number of clusters. These curves were used to compute the unsupervised clustering metric (ADBI) results presented in SS3.2. As shown in the figure, these curves do not preserve the relative ranking we find in the corresponding learning curves.

\begin{table}
\begin{tabular}{p{113.8pt} p{34.1pt} p{34.1pt} p{34.1pt}} \hline \hline Dataset & Size & Prior & Task \\ \hline IMDB & 50K & 50\% & sentiment toxicity \\ WikiToxic & 224K & 9\% & toxicity \\ Sentiment140 & 1.6M & 50\% & sentiment sentiment \\ CivilComments & 2M & 8\% & toxic behav. \\ \hline \hline \end{tabular}
\end{table}
Table 3: Datasets statistics with the number of samples, target (positive) class prior, and classification task.

 

Figure 3: Learning curves (left), task hierarchical alignment curves (center), and DBI curves (right) for all the datasets: IMDB, WikiToxic, Sentiment140, and CivilComments.

 



**ACL 2023 Responsible NLP Checklist+**

Footnote †: margin: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.



C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _3, A.2_
* C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _3, A.2_
* C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _A.2_
* D **Did you use human annotators (e.g., crowdworkers) or research with human participants?** _Left blank._
* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _No response._
* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _No response._
* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _No response._
* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _No response._
* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _No response._