SlowGuess/NAACL_2021_OCR huggingface dataset:
DIRECTPROBE: Studying Representations without Classifiers
Yichu Zhou
flyaway@cs.utah.edu
Understanding how linguistic structure is encoded in contextualized embedding could help explain their impressive performance across NLP. Existing approaches for probing them usually call for training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation's goodness. In this work, we argue that doing so can be unreliable because different representations may need different classifiers. We develop a heuristic, DIRECTPROBE, that directly studies the geometry of a representation by building upon the notion of a version space for a task. Experiments with several linguistic tasks and contextualized embeddings show that, even without training classifiers, DIRECTPROBE can shine light into how an embedding space represents labels, and also anticipate classifier performance for the representation.
Distributed representations of words (e.g., Peters et al., 2018; Devlin et al., 2019) have propelled the state-of-the-art across NLP to new heights. Recently, there is much interest in probing these opaque representations to understand the information they bear (e.g., Kovaleva et al., 2019; Conneau et al., 2018; Jawahar et al., 2019). The most commonly used strategy calls for training classifiers on them to predict linguistic properties such as syntax, or cognitive skills like numeracy (e.g. Kassner and Schütze, 2020; Perone et al., 2018; Yaghoobzadeh et al., 2019; Krasnowska-Kieraś and Wróblewska, 2019; Wallace et al., 2019; Pruksachatkun et al., 2020). Using these classifiers, criteria such as accuracy or model complexity are used to evaluate the representation quality for the task (e.g. Goodwin et al., 2020; Pimentel et al., 2020a; Michael et al., 2020).
Such classifier-based probes are undoubtedly useful to estimate a representation's quality for a
task. However, their ability to reveal the information in a representation is occluded by numerous factors, such as the choice of the optimizer and the initialization used to train the classifiers. For example, in our experiments using the task of preposition supersense prediction (Schneider et al., 2018), we found that the accuracies across different training runs of the same classifier can vary by as much as  $\sim 8\%$ ! (Detailed results can be found in Appendix F.)
Indeed, the very choice of a classifier influences our estimate of the quality of a representation. For example, one representation may achieve the best classification accuracy with a linear model, whereas another may demand a multi-layer perceptron for its non-linear decision boundaries. Of course, enumerating every possible classifier for a task is untenable. A common compromise involves using linear classifiers to probe representations (Alain and Bengio, 2017; Kulmizev et al., 2020), but doing so may mischaracterize representations that need non-linear separators. Some work recognizes this problem (Hewitt and Liang, 2019) and proposes to report probing results for at least logistic regression and a multi-layer perceptron (Eger et al., 2019), or to compare the learning curves between multiple controls (Talmor et al., 2020). However, the success of these methods still depends on the choices of classifiers.
In this paper, we pose the question: Can we evaluate the quality of a representation for an NLP task directly without relying on classifiers as a proxy?
Our approach is driven by a characterization of not one, but all decision boundaries in a representation that are consistent with a training set for a task. This set of consistent (or approximately consistent) classifiers constitutes the version space for the task (Mitchell, 1982), and includes both simple (e.g., linear) and complex (e.g., non-linear) classifiers for the task. However, perfectly characterizing the version space for a problem presents compu
tational challenges. To develop an approximation, we note that any decision boundary partitions the underlying feature space into contiguous regions associated with labels. We present a heuristic approach called DIRECTPROBE, which builds upon hierarchical clustering to identify such regions for a given task and embedding.
The resulting partitions allow us to directly probe the embeddings via their geometric properties. For example, distances between these regions correlate with the difficulty of learning with the representation: larger distances between regions of different labels indicates that there are more consistent separators between them, and imply easier learning, and better generalization of classifiers. Further, by assigning test points to their closest partitions, we have a parameter-free classifier as a side effect, which can help benchmark representations without committing to a specific family of classifiers (e.g., linear) as probes.
Our experiments study five different NLP tasks that involve syntactic and semantic phenomena. We show that our approach allows us to ascertain, without training a classifier, (a) if a representation admits a linear separator for a dataset, (b) how different layers of BERT differ in their representations for a task, (c) which labels for a task are more confusable, (d) the expected performance of the best classifier for the task, and (e) the impact of fine-tuning.
In summary, the contributions of this work are:
1. We point out that training classifiers as probes is not reliable, and instead, we should directly analyze the structure of a representation space.
2. We formalize the problem of evaluating representations via the notion of version spaces and introduce DIRECTPROBE, a heuristic method to approximate it directly which does not involve training classifiers.
3. Via experiments, we show that our approach can help identify how good a given representation will be for a prediction task. $^{1}$
2 Representations and Learning
In this section, we will first briefly review the relationship between representations and model learning. Then, we will introduce the notion of  $\epsilon$ -version spaces to characterize representation quality.
Figure 1: An illustration of the total work in a learning problem, visualized as the distance traveled from the input bar to the output bar. Different representations  $E_{i}$  and different classifiers  $h_{i}$  provide different effort.
2.1 Two Sub-tasks of a Learning Problem
The problem of training a predictor for a task can be divided into two sub-problems: (a) representing data, and (b) learning a predictor (Bengio et al., 2013). The former involves transforming input objects  $x$  — words, pairs of words, sentences, etc. — into a representation  $E(x) \in \mathcal{R}^n$  that provides features for the latter. Model learning builds a classifier  $h$  over  $E(x)$  to make a prediction, denoted by the function composition  $h(E(x))$ .
Figure 1 illustrates the two sub-tasks and their roles in figuratively "transporting" an input  $x$  towards a prediction with a probability of error below a small  $\epsilon$ . For the representation  $E_{1}$ , the best classifier  $h_{1}$  falls short of this error requirement. The representation  $E_{4}$  does not need an additional classifier because it is identical to the label. The representations  $E_{2}$  and  $E_{3}$  both admit classifiers  $h_{2}$  and  $h_{3}$  that meet the error threshold. Further, note, that  $E_{3}$  leaves less work for the classifier than  $E_{2}$ , suggesting that it is a better representation as far as this task is concerned.
This illustration gives us the guiding principle for this work:
The quality of a representation  $E$  for a task is a function of both the performance and the complexity of the best classifier  $h$  over that representation.
Two observations follow from the above discussion. First, we cannot enumerate every possible classifier to find the best one. Other recent work, such
2In addition to performance and complexity of the best classifier, other aspects such as sample complexity, the stability of learning, etc are also important. We do not consider them in this work: these aspects are related to optimization and learnability, and more closely tied to classifier-based probes.
1DIRECTPROBE can be downloaded from https://github.com/utahnlp/DirectProbe.
as that of Xia et al. (2020) make a similar point. Instead, we need to resort to an approximation to evaluate representations. Second, trivially, the best representation for a task is identical to an accurate classifier; in the illustration in Figure 1, this is represented by  $E_4$ . However, such a representation is over-specialized to one task. In contrast, learned representations like BERT promise task-independent representations that support accurate classifiers.
2.2  $\epsilon$ -Version Spaces
Given a classification task, we seek to disentangle the evaluation of a representation  $E$  from the classifiers  $h$  that are trained over it. To do so, the first step is to characterize all classifiers supported by a representation.
Classifiers are trained to find a hypothesis (i.e., a classifier) that is consistent with a training set. A representation  $E$  admits a set of such hypotheses, and a learner chooses one of them. Consider the top-left example of Figure 2. There are many classifiers that separate the two classes; the figure shows two linear  $(h_1$  and  $h_2$ ) and one non-linear  $(h_3$  ) example. Given a set  $\mathcal{H}$  of classifiers of interest, the subset of classifiers that are consistent with a given dataset represents the version space with respect to  $\mathcal{H}$  (Mitchell, 1982). To account for errors or noise in data, we define an  $\epsilon$ -version space: the set of hypothesis that can achieve less than  $\epsilon$  error on a given dataset.
Let us formalize this definition. Suppose  $\mathcal{H}$  represents the whole hypothesis space consisting of all possible classifiers  $h$  of interest. The  $\epsilon$ -version space  $V_{\epsilon}(\mathcal{H}, E, D)$  expressed by a representation  $E$  for a labeled dataset  $D$  is defined as:
$$
V _ {\epsilon} (\mathcal {H}, E, D) \triangleq \{h \in \mathcal {H} \mid e r r (h, E, D) \leq \epsilon \} \tag {1}
$$
where err represents training error.
Note that the  $\epsilon$ -version space  $V_{\epsilon}(\mathcal{H},E,D)$  is only a set of functions and does not involve any learning. However, understanding a representation requires examining its  $\epsilon$ -version space—a larger one would allow for easier learning.
In previous work, the quality of a representation  $E$  for a task represented by a dataset  $D$  is measured via properties of a specific  $h \in V_{\epsilon}(\mathcal{H}, E, D)$ , typically a linear probe. Commonly measured properties include generalization error (Kim et al., 2019), minimum description length of labels (Voita and Titov, 2020) and complexity (Whitney et al., 2020).
Figure 2: Using the piecewise linear functions to mimic decision boundaries on two different cases. The solid lines on the left are the real decision boundaries for a binary classification problem. The dashed lines in the middle are the piecewise linear functions used to mimic these decision boundaries. The gray area is the region that a separator must cross. The connected points in the right represent the convex regions that the piecewise linear separators lead to.
Instead, we seek to directly evaluate the  $\epsilon$ -version space of a representation for a task, without committing to a restricted set of probe classifiers.
3 Approximating an  $\epsilon$ -Version Space
Although the notion of an  $\epsilon$ -version space is well defined, finding it for a given representation and task is impossible because it involves enumerating all possible classifiers. In this section, we will describe a heuristic to approximate the  $\epsilon$ -version space. We call this approach as DIRECTPROBE.
3.1 Piecewise Linear Approximation
Each classifier in  $V_{\epsilon}(\mathcal{H}, E, D)$  is a decision boundary in the representation space that separates examples with different labels (see Figure 2, left). The decisions of a classifier can be mimicked by a set of piecewise linear functions. Figure 2 (middle) shows two examples. At the top is the simple case with linearly separable points. At the bottom is a more complicated case with a circular separator. The set of the piecewise linear function that matches its decisions needs at least three lines.
The ideal piecewise linear separator partitions training points into groups, each of which contains points with exactly one label. These groups can be seen as defining convex regions in the embedding space (see Figure 2, left). Any classifier in  $V_{\epsilon}(\mathcal{H},E,D)$  must cross the regions between the groups with different labels; these are the regions that separate labels from each other, as shown in
the gray areas in Figure 2 (middle). Inspired by this, we posit that these regions between groups with different labels, and indeed the partitions themselves, offer insight into  $V_{\epsilon}(\mathcal{H},E,D)$ .
3.2 Partitioning Training Data
Although finding the set of all decision boundaries remain hard, finding the regions between convex groups that these piecewise linear functions splits the data into is less so. Grouping data points in this fashion is related to a well-studied problem, namely clustering, and several recent works have looked at clustering of contextualized representations (Reimers et al., 2019; Aharoni and Goldberg, 2020; Gupta et al., 2020). In this work, we have a new clustering problem with the following criteria: (i) All points in a group have the same label. We need to ensure we are mimicking the decision boundaries. (ii) There are no overlaps between the convex hulls of each group. If convex hulls of two groups do not overlap, there must exist a line that can separates them, as guaranteed by the hyperplane separation theorem. (iii) Minimize the number of total groups. Otherwise, a simple solution is that each data point becomes a group by itself.
Note that the criteria do not require all points of one label to be grouped together. For example, in Figure 2 (bottom right), points of the circle (i.e.,  $\circ$ ) class are in three subgroups.
To summarize what we have so far: we transformed the problem of finding the  $\epsilon$ -version space into a clustering problem with specific criteria. Next, let us see a heuristic for partitioning the training set into clusters based on these criteria.
3.3 A Heuristic for Clustering
To find clusters as described in §3.2, we define a simple bottom-up heuristic clustering strategy that forms the basis of DIRECTPROBE (Algorithm 1). In the beginning, each example  $x_{i}$  with label  $y_{i}$  in a training set  $D$  is a cluster  $C_i$  by itself. In each iteration, we select the closest pair of clusters with the same label and merge them (lines 4, 5). If the convex hull of the new cluster does not overlap with all other clusters<sup>3</sup>, we keep this new cluster (line 9). Otherwise, we flag this pair (line 7) and
choose the next closest pair of clusters. We repeat these steps till no more clusters can be merged.
Algorithm 1 DIRECTPROBE: Bottom-up Clustering
Input: A dataset  $D$  with labeled examples  $(x_{i},y_{i})$ .
Output: A set of clusters of points  $\mathcal{C}$ .
1: Initialize  $C_i$  as the cluster for each  $(x_i, y_i) \in D$
2:  $\mathcal{C} = \{C_0,C_1,\dots \}$ $B = \emptyset$
3: repeat
4: Select the closest pair  $(C_i, C_j) \in \mathcal{C}$  which have the same label and is not flagged in  $B$ .
5:  $S \leftarrow C_i \cup C_j$
6: if convex hull of  $S$  overlaps with other elements of  $\mathcal{C}$  then
7:  $B\gets B\cup \{(C_i,C_j)\}$
8: else
9: Update  $\mathcal{C}$  by removing  $C_i, C_j$  and adding  $S$
10: end if
11: until no more pairs can be merged
12: return  $\mathcal{C}$
We define the distance between clusters  $C_i$  and  $C_j$  as the Euclidean distance between their centroids. Although Algorithm 1 does not guarantee the minimization criterion in §3.2 since it is greedy heuristic, we will see in our experiments that, in practice, it works well.
Overlaps Between Convex Hulls A key point of Algorithm 1 is checking if the convex hulls of two sets of points overlap (line 6). Suppose we have two sets  $C = \{x_1^C,\ldots ,x_n^C\}$  and  $C^\prime =$ $\{x_1^{C'},\dots,x_m^{C'}\}$  . We can restate this as the problem of checking if there is some vector  $w\in \Re^n$  and a number  $b\in \Re$  such that:
$$
\begin{array}{l} \forall x _ {i} ^ {C} \in C, \quad w ^ {\top} E (x _ {i} ^ {C}) + b \geq 1, \\ \forall x _ {j} ^ {C ^ {\prime}} \in C ^ {\prime}, \quad w ^ {\top} E \left(x _ {j} ^ {C ^ {\prime}}\right) + b \leq - 1. \tag {2} \\ \end{array}
$$
where  $E$  is the representation under investigation.
We can state this problem as a linear program that checks for feasibility of the system of inequalities. If the LP problem is feasible, there must exist a separator between the two sets of points, and they do not overlap. In our implementation, we use the Gurobi optimizer (Gurobi Optimization, 2020) to solve the linear programs.4
3.4 Noise: Controlling  $\epsilon$
Clusters with only a small number of points could be treated as noise. Geometrically, a point in the neighborhood of other points with different labels
3Note that "all other clusters" here means the other clusters with different labels. There is no need to prohibit overlap between clusters with the same label since they might be merged in the next iterations.
4 Algorithm 1 can be made faster by avoiding unnecessary calls to the solver. Appendix A gives a detailed description of these techniques, which are also incorporated in the code release.
could be thought of as noise. Other clusters can not merge with it because of the no-overlap constraint. As a result, such clusters will have only a few points (one or two in practice). If we want zero error rate on the training data, we can keep these noise points; if we allow a small error rate  $\epsilon$ , then we can remove these noise clusters. In our experiments, for simplicity, we keep all clusters.
4 Representations, Tasks and Classifiers
Before looking at the analysis offered by the partitions obtained via DIRECTPROBE in §5, let us first enumerate the English NLP tasks and representations we will encounter.
4.1 Representations
Our main experiments focus on  $\mathrm{BERT}_{\mathrm{base,cased}}$ , and we also show additional analysis on other contextual representations: ELMo (Peters et al., 2018) $^5$ ,  $\mathrm{BERT}_{\mathrm{large,cased}}$  (Devlin et al., 2019), RoBERTa $_{\mathrm{base}}$  and RoBERTa $_{\mathrm{large}}$  (Liu et al., 2019b). We refer the reader to the Appendix C for further details about these embeddings.
We use the average of subword embeddings as the token vector for the representations that use subwords. We use the original implementation of ELMo, and the HuggingFace library (Wolf et al., 2020) for the others.
We conduct our experiments on five NLP tasks that cover the varied usages of word representations (token-based, span-based, and token pairs) and include both syntactic and semantic prediction problems. The Appendix D has more details about the tasks to help with replication.
Preposition supersense disambiguation represents a pair of tasks, involving classifying a preposition's semantic role (SS-role) and semantic function (SS-function). Following the previous work (Liu et al., 2019a), we only use the single-token prepositions in the Streusle v4.2 corpus (Schneider et al., 2018).
Part-of-speech tagging (POS) is a token level prediction task. We use the English portion of the parallel universal dependencies treebank (ud-pud Nivre et al., 2016).
Semantic relation (SR) is the task of predicting the semantic relation between pairs of nominals.
We use the dataset of semeval2010 task 8 (Hendrickx et al., 2010). To represent the pair of nominals, we concatenate their embeddings. Some nominals could be spans instead of individual tokens, and we represent them via the average embedding of the tokens in the span.
Dependency relation (DEP) is the task of predicting the syntactic dependency relation between a token  $w_{head}$  and its modifier  $w_{mod}$ . We use the universal dependency annotation of the English web treebank (Bies et al., 2012). As with semantic relations, to represent the pair of tokens, we concatenate their embeddings.
4.3 Classifier Accuracy
The key starting point of this work is that restricting ourselves to linear probes may be insufficient. To validate the results of our analysis, we evaluate a large collection of classifiers—from simple linear classifiers to two-layers neural networks—for each task. For each one, we choose the best hyperparameters using cross-validation. From these classifiers, we find the best test accuracy of each task and representation. All classifiers are trained with the scikit-learn library (Pedregosa et al., 2011). To reduce the impact of randomness, we trained each classifier 10 times with different initializations, and report their average accuracy. The Appendix E summarizes the best classifiers we found and their performance.
DIRECTPROBE helps partition an embedding space for a task, and thus characterize its  $\epsilon$ -version space. Here, we will see that these clusters do indeed characterize various linguistic properties of the representations we consider.
5.1 Number of Clusters
The number of clusters is an indicator of the linear separability of representations for a task. The best scenario is when the number of clusters equals the number of labels. In this case, examples with the same label are placed close enough by the representation to form a cluster that is separable from other clusters. A simple linear multi-class classifier can fit well in this scenario. In contrast, if the number of clusters is more than the number of labels, then some labels are distributed across multiple clusters (as in Figure 2, bottom). There must be a non-linear decision boundary. Consequently,
For simplicity, we use the equally weighted three layers of ELMo in our experiments.
Embedding	Linear SVM Training Accuracy	#Clusters
BERTbase,cased	100	17
BERTlarge,cased	100	17
RoBERTa base	100	17
RoBERTa large	99.97	23
ELMo	100	17
Table 1: Linearity experiments on POS tagging task. Our POS tagging task has 17 labels in total. Both linear SVM and the number of clusters suggest that RoBERTa large is non-linear while others are all linear, which means the best classifier for RoBERTa large is not a linear model. More details can be found in Appendix E.
this scenario calls for a more complex classifier, e.g., a multi-layer neural network.
In other words, using the clusters, and without training a classifier, we can answer the question: can a linear classifier fit the training set for a task with a given representation?
To validate our predictions, we use the training accuracy of a linear SVM (Chang and Lin, 2011) classifier. If a linear SVM can perfectly fit (100% accuracy) a training set, then there exist linear decision boundaries that separate the labels. Table 1 shows the linearity experiments on the POS task, which has 17 labels in total. All representations except RoBERTalarge have 17 clusters, suggesting a linearly separable space, which is confirmed by the SVM accuracy. We conjecture that this may be the reason why linear models usually work for BERT-family models. Of course, linear separability does not mean the task is easy or that the best classifier is a linear one. We found that, while most representations we considered are linearly separable for most of our tasks, the best classifier is not always linear. We refer the reader to Appendix E for the full results.
5.2 Distances between Clusters
As we mentioned in §3.1, a learning process seeks to find a decision boundary that separates clusters with different labels. Intuitively, a larger gap between them would make it easier for a learner to find a suitable hypothesis  $h$  that generalizes better.
We use the distance between convex hulls of clusters as an indicator of the size of these gaps. We note that the problem of computing the distance between convex hulls of clusters is equivalent to finding the maximum margin separator between them. To find the distance between two clusters, we train a linear SVM (Chang and Lin, 2011) that