SlowGuess/ACL_2020_OCR huggingface dataset:
Fynn Schroder
Language Technology Group
fschroeder@informatik.uni-hamburg.de
Chris Biemann
biemann@informatik.uni-hamburg.de
Multi-task learning (MTL) and transfer learning (TL) are techniques to overcome the issue of data scarcity when training state-of-the-art neural networks. However, finding beneficial auxiliary datasets for MTL or TL is a time- and resource-consuming trial-and-error approach. We propose new methods to automatically assess the similarity of sequence tagging datasets to identify beneficial auxiliary data for MTL or TL setups. Our methods can compute the similarity between any two sequence tagging datasets, i.e. they do not need to be annotated with the same tagset or multiple labels in parallel. Additionally, our methods take tokens and their labels into account, which is more robust than only using either of them as an information source, as conducted in prior work. We empirically show that our similarity measures correlate with the change in test score of neural networks that use the auxiliary dataset for MTL to increase the main task performance. We provide an efficient, open-source implementation. $^{1}$
State-of-the-art neural networks usually require large amounts of training data and vast computational resources. Especially for low-resource tasks, data scarcity is the main issue hampering the training of robust models. By leveraging multitask learning or transfer learning, auxiliary data can be incorporated into the training to boost the main task performance. Finding suitable auxiliary datasets for these cases is a time- and resource-consuming trial-and-error approach, because there can be plenty of plausible auxiliary datasets that could help to learn the main task. For a proper evaluation of different auxiliary datasets, hyperparameter search and training runs with multiple random seeds have to be performed for each auxiliary
dataset individually. Thus, the process takes even longer and uses even more computational resources. We propose methods to shorten this trial-and-error approach by computing the similarity between any two sequence tagging datasets. Based on the similarity, suitable datasets can be quickly selected to be used as auxiliary training data for multi-task or transfer learning.
Our contributions are a family of novel methods to compute the similarity of sequence tagging datasets, where the similarity values correlate with the change in multi-task learning performance when using one dataset as auxiliary data for training the other. We evaluate our methods in experiments with five part-of-speech (POS) tagging, nine named-entity recognition (NER) and three argumentation mining (AM) datasets. Our similarity measures allow for comparison both datasets for the same and different tasks, not requiring the same set of labels on target and auxiliary dataset. The calculated similarity scores can be used to predict which dataset will be beneficial as auxiliary training data for multi-task training in order to shorten the search process.
2.1 Neural multi-task and transfer learning
Multi-task learning (MTL) is a technique to learn multiple tasks jointly (Caruana, 1997). Depending on the setting, either all tasks are equally important, or only the performance on the main task is of interest, which shall be improved with additional training data. MTL has been successfully applied in natural language processing for various sequence tagging tasks (Søgaard and Goldberg, 2016; Bjerva et al., 2016; Plank et al., 2016; Martínez Alonso and Plank, 2017; Kaiser et al., 2017; Bingel and Søgaard, 2017; Augenstein and Søgaard, 2017; Kim et al., 2017; Yang et al., 2017; Changpinyo
github.com/uhh-lt/seq-tag-sim
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2971-2985
et al., 2018; Liu et al., 2018; Schulz et al., 2018). These approaches use hard parameter sharing in the hidden layers of neural learning architectures, where the same weights are updated from several tasks. The majority of works combined a main task with a single, supervised auxiliary task.
In transfer learning, a model is pre-trained on an auxiliary dataset to increase the main task performance. Howard and Ruder (2018) showed knowledge transfer based on large-scale language modeling. Before the breakthrough with BERT (Devlin et al., 2019), only partial knowledge transfer via word embeddings such as word2vec (Mikolov et al., 2013) or ELMo (Ilic et al., 2018) was utilized.
2.2 Effect of auxiliary task similarity
In theory, auxiliary tasks can have various relationships to the main task (Ruder, 2017). In practice, the most common choice is to use a "somehow" related task. Caruana (1997) argues that tasks are similar if the same features are used for making predictions. Baxter (2000) suggests similar tasks should have the same inductive bias. Ben-David and Schuller (2003) indicate that tasks originating from the same probability distribution are similar and perform well in an MTL setting. No universal measure for task similarity exists, but it is needed to select tasks to prefer for training (Ruder, 2017).
Although MTL is frequently applied in recent work, few elaborate on the effect of task and dataset similarity. Recent work on neural MTL found different hints regarding task similarity that are only applicable to a specific scenario. Kim et al. (2017) performed MTL on POS tagging across 14 languages and found that language similarity seems to correlate with MTL performance. Yang et al. (2017) worked on common tasks with artificially reduced datasets. They attribute the degree of performance increase to label abundance for the main task, dataset similarity and number of shared parameters. Changpinyo et al. (2018) compared eleven tasks and observed that some tasks increase the performance in most cases, while tasks with a small tagset decreased the main task performance. In contrast, Martínez Alonso and Plank (2017) show results that auxiliary tasks with few labels and a uniform label distribution perform better for MTL in neural sequence tagging: Auxiliary tasks having many labels or high entropy harm the main task performance. While Ruder et al. (2019) confirm these findings, Bjerva (2017) found no evi
dence of label entropy correlating with MTL performance. Martínez Alonso and Plank (2017) found a difference between two POS datasets when used as auxiliary data because converting one to another tagset changes the effect of MTL significantly.
Kim et al. (2015) propose a method using label embeddings to map labels from auxiliary datasets to the target tagset so that MTL can be treated as single-task learning (STL) with an increased amount of training data. Bingel and Søgaard (2017) predict MTL performance from dataset and STL learning features and found the learning curve to be much more important. From the dataset features, the number of labels on the main task and the auxiliary label entropy showed predictive potential.
Most similar to our approach is the work of Bjerva (2017), who estimates the effect of an auxiliary task in MTL with information-theoretic measures. As the method requires the same datasets to be tagged with multiple tasks in parallel, at least one task must be automatically taggable with almost perfect results. He shows a correlation of conditional entropy and mutual information with a change in accuracy compared to STL. Results on the semantic task of Bjerva et al. (2016); Martínez Alonso and Plank (2017) indicate that mutual information for helpful auxiliary tasks is higher than for harmful tasks.
Augenstein et al. (2018) propose an architecture that learns label embeddings for natural language classification tasks and find that label embeddings indicate gains or harms of MTL. Ruder et al. (2019) correlate task properties with performance differences and learned meta-network parameters of their proposed sluice networks. They find that MTL gains are higher for smaller training datasets and that sluice networks learn to share more in case of higher variance in the training data.
Opposed to previous approaches, our methods can compare same-task datasets and are not restricted to datasets with parallel labels. As our experiments in Section 5 require these properties, previous approaches are not applicable and thus not comparable. Next, we will introduce information-theoretic measures that build the foundation for our dataset similarity measures proposed in Section 4.
3 Information-theoretic clustering comparison measures
Entropy is a measure of the uncertainty of a random variable. The entropy  $H(X)$  of a discrete random
variable  $X$  with alphabet  $\mathcal{X}$  is defined as
$$
H (X) = - \sum_ {x \in \mathcal {X}} p (x) \log_ {2} p (x) \tag {1}
$$
where  $p(x)$  is the probability mass function  $p(x) = \operatorname*{Pr}\{X = x\}, x \in \mathcal{X}$ . It is 0 when  $p = 0$  or 1 and maximal when  $p = \frac{1}{|\mathcal{X}|}$  (uniform distribution) with an upper bound of  $H(X) \leq \log_2|X|$ .
Joint entropy  $H(X,Y)$  extends entropy from a single to two random variables. For a pair of discrete random variables  $(X,Y)$  with a joint probability distribution  $p(x,y)$ , it is defined as
$$
H (X, Y) = - \sum_ {x \in \mathcal {X}} \sum_ {y \in \mathcal {Y}} p (x, y) \log_ {2} p (x, y). \tag {2}
$$
Mutual information (MI)  $I(X;Y)$  describes the amount of information one random variable  $X$  contains about another  $Y$ . It is a symmetric measure of range  $[0,\min \{H(X),H(Y)\}]$  defined as
$$
I (X; Y) = \sum_ {x \in \mathcal {X}} \sum_ {y \in \mathcal {Y}} p (x, y) \log_ {2} \frac {p (x , y)}{p (x) p (y)} \tag {3}
$$
with probability mass functions  $p(x), p(y)$  and a joint probability mass function  $p(x, y)$ . For a detailed description of entropy, mutual information and information theory in general, please refer to Cover and Thomas (2006).
A clustering  $C$  is a way to partition a dataset  $D$  into non-overlapping subsets  $\{c_1, c_2, \ldots\}$  together containing all  $N$  items of  $D$ . Comparing clusterings requires a measure to determine the quality of a clustering according to another clustering, e.g. the ground truth. Such a measure should quantify the amount of information shared between both clusterings. (Vinh et al., 2010)
Information-theoretic clustering comparison measures are based on a solid mathematical foundation from information theory and can work with non-linear similarities. They have become popular by the works of Strehl and Ghosh (2003) and Meilă (2005).
Mutual information measures the information shared between two clusterings  $C$  and  $C'$ . A higher MI signals a greater help in predicting the cluster labels in  $C$  with information from  $C'$ . Several normalized mutual information variants can be derived:
$$
N M I _ {j o i n t} = \frac {I \left(C ; C ^ {\prime}\right)}{H \left(C , C ^ {\prime}\right)} \tag {4}
$$
$$
N M I _ {m a x} = \frac {I \left(C ; C ^ {\prime}\right)}{\operatorname* {m a x} \left(H (C) , H \left(C ^ {\prime}\right)\right)} \tag {5}
$$
Analogously to  $NMI_{max}$ , there are  $NMI_{sum}$ ,  $NMI_{sqrt}$  and  $NMI_{min}$  that use entropy sums, square root of the entropy products or minimum of both entropy values as a normalization factor (Kvalseth, 1987; Strehl and Ghosh, 2003; Yao, 2003; Liu et al., 2008). They are all bounded in [0, 1], equaling 0 when two clusterings share no information at all, i.e. are fully independent and 1 when two clusterings are identical.
According to Vinh et al. (2010),  $NMI_{max}$  and  $NMI_{joint}$  satisfy the highest number of theoretical properties desirable among the clustering comparison measures. They prove that only the unit complements of both measures satisfy the metric property (positive definiteness, symmetry and triangle inequality). While all measures satisfy the normalization property, none conform to the constant baseline property unless the number of items  $N$  is large, compared to the number of clusters.
The high-level idea of our dataset similarity measures is the following: Words and labels from one dataset are correlated with the words and their labels from another dataset to create a probabilistic mapping between both label sets. Either an exact string matching or a fuzzy matching based on word embedding representations can be used. The dataset similarity is measured via the quality of this label mapping.
4.1 Casting label similarity as a clustering comparison problem
Transforming the problem of token-label dataset similarity to a clustering comparison problem allows reusing existing clustering comparison measures. A clustering represents one label set, and each label is a cluster within the clustering, i.e. all tokens having the same label belong to one cluster.
A contingency table, also called a confusion matrix, is a handy tool to compare clusterings. Let us assume that a dataset  $D$  is annotated with two labels in parallel from two tasks  $T$  and  $T'$  with arbitrary label sets  $L$  and  $L'$ . The comparison of  $L$  with  $L'$  on  $D$  can be transformed into a clustering comparison problem. The clusters for  $T$  are the labels  $l_1, l_2, \ldots, l_N$  when the label set  $L$  has  $N$  different labels in total. The clusters for  $T'$  are labeled analogously  $l_1', l_2', \ldots, l_M'$  for the  $M$  labels in the set  $L'$ . Table 1 shows the resulting contingency table for the described setting. The values  $c_{xy}$  are
the counts how many tokens are in the dataset that are labeled as / belong to cluster  $l_{x}$  in task  $T$  and simultaneously  $l_{y}^{\prime}$  in the task  $T^{\prime}$ .<sup>2</sup>
l1'	l2'	...	lM'	Σ
l1	c11	c12	...	c1M	c1.
l2	c21	c22	...	c2M	c2.
...	...	...	...	...
lN	cN1	cN2	...	cNM	cN.
Σ	c.1	c.2	...	c.M	c
Table 1: Contingency table for a comparison of label sets  $L$  and  ${L}^{\prime }$  with  $N$  and  $M$  unique labels
Based on the counts in the contingency table, information-theoretic measures such as (joint) entropy or mutual information can be calculated. Because the probability mass functions  $p(x), p(y)$  and  $p(x,y)$  are unknown for the label sets  $L$  and  $L'$  in dataset  $D$ , the probabilities are approximated by the relative frequencies of the label pairs. The entropy of both label sets has to be taken into account to know whether the tasks  $T$  and  $T'$  are similar, i.e. a normalized mutual information variant shown in Equations 4 and 5 has to be used. With the notation in Table 1, the  $NMI_{joint}$  definition becomes
$$
\begin{array}{l} N M I (L, L ^ {\prime}) _ {j o i n t} = \frac {I (L ; L ^ {\prime})}{H (L , L ^ {\prime})} \\ = \frac {\sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} \frac {c _ {i j}}{c} \log_ {2} \left(\frac {c _ {i j} c}{c _ {i} . c _ {j}}\right)}{- \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} \frac {c _ {i j}}{c} \log_ {2} \left(\frac {c _ {i j}}{c}\right)}. \tag {6} \\ \end{array}
$$
The other measures can be changed analogously.
Next, we show how to transform label similarity to clustering comparison without being restricted to datasets annotated in parallel with both label sets.
4.2 Obtaining label pairs from datasets
To compare two datasets, one of the datasets can be tagged automatically with the other task's labels as proposed by Bjerva (2017). However, a comparison is only possible if at least one of the tasks can be tagged automatically with near-perfect accuracy. While the necessary performance-level has been reached for a few simple tasks, the state-of-the-art performance on most tasks seems insufficient for this purpose. Further, two datasets of the same task, e.g. two NER datasets with the same tagset, cannot be meaningfully compared when tagged automatically. We propose two approaches to lift the
restrictions on the datasets and tasks. The solutions enable a comparison of arbitrary task and dataset combinations.
4.2.1 Text overlap
If a manually defined one-to-one mapping from labels of one dataset to another one exists, datasets can be compared to each other using this label mapping function, because it produces a dataset with parallel label sets. While mapping a fine-grained label set to a coarse label set is possible, it is unclear how to map a coarse label to finer sub-labels.
The text overlap approach implicitly generates a label mapping from the token-label pairs of both datasets. This has the advantage of being independent of external knowledge and enabling a probabilistic mapping from coarse to fine-grained label sets specific to the datasets. Tokens are aggregated so that a token is associated with the number of times it has been tagged with each label. Only tokens occurring in both datasets can be used to fill in the counts of a contingency table. By looking only at the intersection of tokens occurring in both datasets, a new virtual dataset is created, where each token is tagged with two labels. For each token, the count at the position  $(l_i, l_j')$  in the contingency table is increased by a combination of the number of times the current token was tagged with labels  $l_i$  and  $l_j'$ . With the additive method to fill a contingency table, label counts for words from both datasets are added because they are viewed as multiple instances from one dataset.
An alternative to addition is to use multiplication to combine the counts for matching words. The counts for each label combination are multiplied and added at the corresponding position in the contingency table. An effect of this approach is that words being frequent in both datasets contribute more to be counts. There are more possible schemes on how to combine the raw counts from two datasets into a mutual contingency table. Similarity measures such as NMI can be computed on any contingency table obtained from these methods.
An advantage of the text overlap approach is that it is fast because it only involves text processing and a few counts. The downside is that an identical dataset can only be identified with  $100\%$  similarity if each word always has the same label. Another issue is that only a fraction of each dataset is used
3 Illustrating examples are provided in Appendix A.2
2 Illustrating examples are provided in Appendix A.1
for the actual comparison. As the plain text overlap approach does not consider the ratio of shared vocabulary, it is possible to have a "false positive", i.e. a high similarity is reported for two datasets although they share only one word. To fix this, we combine the NMI value and the ratio of shared vocabulary (SV) via the harmonic mean into our text overlap (TO) measure
$$
T O = \frac {2 \cdot N M I \cdot S V}{N M I + S V} \tag {7}
$$
with the shared vocabulary
$$
S V = \frac {\left| V \cap V ^ {\prime} \right|}{\left| V \cup V ^ {\prime} \right|} \tag {8}
$$
where  $V$  and  $V^{\prime}$  are the sets of all unique words in the two datasets  $D$  and  $D^{\prime}$ .
When constructing the contingency table (e.g. Table 1) with the text overlap approach, the sequence information of label-word pairs, i.e. the context, cannot be captured in the counts. With the usage of contextual embeddings, this issue can be mitigated sufficiently.
4.2.2 Vector space similarity
Word embeddings allow representing words in the form of dense vectors within a vector space instead of a specific character sequence in the language's vocabulary. Thus, it is possible to perform mathematical operations on these vectors and compute e.g. the semantic similarity of two words by computing their cosine similarity within the vector space (Elekes et al., 2017). These word vector techniques can be used to tackle the problems of the previously shown text overlap approach.
A first extension allows incorporating words not occurring in both datasets. Vector representations are obtained for each unique word in the datasets. Instead of ignoring words contained only in one dataset, the closest word from the other dataset is chosen via cosine similarity for the pairwise label comparison. The remaining process and similarity measure computation stays the same. $^4$
In the vector space approach, all tokens are compared. For each token, a unique vector representation is obtained via contextual embeddings such as  $ELMo$  (Ilic et al., 2018) or  $BERT$  (Devlin et al., 2019). In order to fill in the counts of a contingency table, each token from one dataset is matched with the most similar vector representation in the other
dataset and the count for the label-pair is increased by the vector space similarity of the two tokens. The usage of contextual embeddings allows to incorporate the sequence information of label-word pairs into the counts. A similarity measure like NMI can be calculated from these counts as before. Identical datasets can be scored with  $100\%$  similarity when the contextual embeddings are able to produce unique vector representations for each token. In general, this method handles ambiguity in language much better as compared to the plain text approach, which should help to improve the similarity comparison between various datasets. Because the process of selecting the closest vector representation from the main dataset to the auxiliary dataset or vice versa can result in different combinations, the counts in the contingency table will be different depending on the direction. Thus, for a symmetric similarity measure like NMI, two scores are obtained. We further combine the forward and backward direction using the harmonic mean into a unified undirectional embedding (UUE) measure:
$$
U U E = \frac {2 \cdot N M I _ {\text {f o r w a r d}} \cdot N M I _ {\text {b a c k w a r d}}}{N M I _ {\text {f o r w a r d}} + N M I _ {\text {b a c k w a r d}}} \tag {9}
$$
The forward and backward NMI in Equation 9 use the same NMI formula and applies it to different counts obtained from the two directions of embeddings comparisons. In our experiments, the actual NMI formula is either  $NMI_{max}$  or  $NMI_{joint}$  due to their desirable theoretical properties.
In this section, experiments will be performed to check whether the similarity of two datasets correlates with the effect on the MTL performance when using the second dataset as auxiliary training data.
5.1 Controlled environment experiments
Before the similarity measures are evaluated together with the MTL performance, we evaluate them independently in a controlled environment. We perform a sanity check by comparing the similarity scores with the intuitive, expected outcome.
Two POS tagging datasets (WSJ, EWT) and two NER datasets (CNLE, ONT) shown in Table 2 will be used to sample three new, non-overlapping datasets each. The samples are named e.g. WSJ-1, WSJ-2, and WSJ-3. Their sizes are equal to  $\frac{1}{6}$ ,  $\frac{2}{6}$  and  $\frac{3}{6}$  of the original number of tokens. Under the assumption that the similarity within samples from
4 Illustrating examples are provided in Appendix A.3
the same original dataset is higher than the similarity between samples from different datasets, the pairwise NMI scores can be qualitatively evaluated.
Figure 1: Pairwise  $NMI_{joint}$  similarity scores (Equation 6) obtained on contingency tables filled with the vector space similarity approach using contextual BERT embeddings. The heat map encodes the values from 0.0 in black to 1.0 in white.
Figure 1 shows the pairwise  $NMI_{joint}$  similarity scores obtained with Equation 6 between these twelve samples. The pairs of identical datasets create a visible diagonal line of maximal similarity. The visible  $3 \times 3$  blocks along the diagonal show high similarity scores and are aligned with comparisons of samples within the same original dataset. Per row or column, the values within these blocks are higher than any other value outside. Thus, the  $NMI_{joint}$  score allows identifying other samples of the same original datasets.
Another interesting property is that the similarity between samples of the two original POS tagging datasets (WSJ, EWT) is higher than the similarity between any POS-NER pair. The same is true the other way around for the NER dataset samples (CNLE, ONT). Hence, the  $NMI_{joint}$  score can be used to distinguish datasets of the same task from others. Note that all four original datasets use different tagsets with a greatly varying number of tags (see Table 2) and that neither the shared vocabulary nor the joint label entropy can be employed to distinguish the POS and NER samples correctly.