SlowGuess/ACL_2020_OCR huggingface dataset:
Computational Linguistics, pages 2066-2081, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Liyuan Liu, Jingbo Shang, Frank F. Xu, Xiang Ren, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower Sequence Labeling with Task-Aware Neural Language Model. In Proceedings of the Thirty-Second Conference on Artificial Intelligence (AAAI-2018), pages 5253-5260, New Orleans, Louisiana, USA.
Zhenqiu Liu, Zhongmin Guo, and Ming Tan. 2008. Constructing tumor progression pathways and biomarker discovery with fuzzy kernel kmeans and dna methylation data. *Cancer informatics*, 6:1-7.
Mitchell P. Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. Penn Treebank 3 LDC99T42. Web Download. Philadelphia: Linguistic Data Consortium.
Héctor Martínez Alonso and Barbara Plank. 2017. When is multitask learning effective? semantic sequence prediction under varying data conditions. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 44-53, Valencia, Spain. Association for Computational Linguistics.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Tackstrom, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92-97, Sofia, Bulgaria. Association for Computational Linguistics.
Marina Meilä. 2005. Comparing clusterings: An axiomatic view. In Proceedings of the 22Nd International Conference on Machine Learning, ICML '05, pages 577-584, Bonn, Germany. ACM.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations (ICLR), Workshop Track Proceedings, Scottsdale, Arizona, USA.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics
(Volume 2: Short Papers), pages 412-418, Berlin, Germany. Association for Computational Linguistics.
Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv:1706.05098.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019. Latent multi-task architecture learning. In Proceedings of the Thirty-Third Conference on Artificial Intelligence (AAAI-2019), pages 4822-4829, Honolulu, Hawaii, USA. Association for the Advancement of Artificial Intelligence.
Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Technology Association Workshop 2015, pages 84-90, Parramatta, Australia.
Claudia Schulz, Steffen Eger, Johannes Daxenberger, Tobias Kahse, and Iryna Gurevych. 2018. Multi-task learning for argumentation mining in low-resource settings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 35-41, New Orleans, Louisiana. Association for Computational Linguistics.
Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 2897-2904, Reykjavik, Iceland. European Language Resources Association (ELRA).
Anders Søgaard and Yoav Goldberg. 2016. Deep multitask learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 231-235, Berlin, Germany. Association for Computational Linguistics.
Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures in persuasive essays. Computational Linguistics, 43(3):619-659.
Alexander Strehl and Joydeep Ghosh. 2003. Cluster ensembles - a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research (JMLR), 3:583-617.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 - Volume 4, CoNLL '03, pages 142-147, Edmonton, Canada. Association for Computational Linguistics.
Nguyen Xuan Vinh, Julien Epps, and James Bailey. 2010. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. Journal of Machine Learning Research (JMLR), 11:2837-2854.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. OntoNotes Release 5.0 LDC2013T19. Web Download. Philadelphia: Linguistic Data Consortium.
Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In 5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings, Toulouse, France.
Yiyu Yao. 2003. Information-Theoretic Measures for Knowledge Discovery and Data Mining, pages 115-136. Springer Berlin Heidelberg, Berlin, Heidelberg.
A.1 Examples for casting label similarity as a clustering comparison problem
Let the dataset  $D$  use simplified named entity recognition (NER) as task  $T$  and part-of-speech (POS) tagging as task  $T^{\prime}$  having the label sets:
$$
\begin{array}{l} L = \{\text {O R G a n i z a t i o n}, \text {P E R s o n}, \text {L O C a t i o n}, \text {O T H e r} \} \\ L ^ {\prime} = \{\text {N N n o u n , V B v e r b , D T d e t e r m i n e r , X o t h e r} \} \\ \end{array}
$$
Let dataset  $D$  contain the following two sentences:
ORG ORG ORG OTH OTH OTH NN NN NN VB DT NN
Walt Disney Productions created the cartoon
OTH PER PER
NN NN NN
character Donald Duck
LOC OTH OTH OTH OTH OTH LOC
NN VB DT X NN X NN
Berlin is a large city in Germany
Table 5 shows the contingency table filled with the counts from both example sentences. The last row resp. column shows the sum of the counts in each column resp. row. The count  $c_{\mathrm{ORG,NN}}$  is three because there are exactly three tokens (Walt Disney Productions) tagged both ORG and NN. Other label-pairs are derived analogously from the remaining tokens of the dataset  $D$ .
With Equation 6, the normalized mutual information can be calculated from the counts in the contingency table. Note that the logarithm is only defined for positive values, but the counts
NN	VB	DT	X	Σ
ORG	3	0	0	0	3
PER	2	0	0	0	2
LOC	2	0	0	0	2
OTH	3	2	2	2	9
Σ	10	2	2	2	16
$c_{ij}$  are often zero. The convention  $0\log (0) = 0$  is used to mitigate this issue because  $x\log (x)\to 0$  when  $x\rightarrow 0$ . The normalized mutual information for the data in Table 5 can now be calculated:  $I(L;L^{\prime}) = 0.437893$  and  $H(L,L^{\prime}) = 2.78064$ . Finally,  $NMI_{joint} = 0.157479$ .
A.2 Examples for the text overlap approach
Below are two example datasets annotated with the reduced POS tagset introduced previously:
(10) VB DT NN X VB DT
Creating an example to explain the
NN VB DT X NN X X
process is an impossible task. To
VB DT NN X NN X NN
process the data , counts of words
X NN VB VB X
and labels are needed.
(11) X VB DT NN X DT X
This is the data for the second
NN X DT NN X VB DT X
dataset . The process to find the right
NN X XX NN VB DT NN
words for this example took a second
Table 6 shows the two Datasets 10 and 11 after the transformation. In the examples, the words process and second are ambiguous without context and thus have multiple labels. Table 7 shows the result of the additive method to combine the label counts from both datasets. The word example occurs once in each dataset and is both times tagged as NN. In the contingency table the count for (NN, NN), i.e. row 2 column 2, is increased by two. The word the occurs two resp. three times in the datasets and is always labeled DT. Consequently, the count in the contingency table at (DT, DT), i.e. row 1 column 1, is increased by five. For process, an issue is that
Table 5: Counts from example dataset  $D$  for comparison of NER and POS tagsets
Word	#	DT	NN	VB	X
example	1	0	1	0	0
to	1	0	0	0	1
the	2	2	0	0	0
process	2	0	1	1	0
is	1	0	0	1	0
.	2	0	0	0	2
data	1	0	1	0	0
words	1	0	1	0	0
(a) Counts for words and their labels in Dataset 10
Word	#	DT	NN	VB	X
is	1	0	0	1	0
the	3	3	0	0	0
data	1	0	1	0	0
.	2	0	0	0	2
process	1	0	1	0	0
to	1	0	0	0	1
words	1	0	1	0	0
example	1	0	1	0	0
(b) Counts for words and their labels in Dataset 11
Table 6: Transformation of word-label pairs to an associated count-based representation. Only words occurring in both datasets are shown.
it has multiple labels in the first dataset: NN and VB. In the second dataset, there is only a single occurrence of process with label NN. The counts in the contingency table are increased by two for the positions (NN, NN) and (VB, NN). However, the single occurrence is now used twice. An improvement is to split the counts by the number of labels in the other dataset, so that the two affected positions are not increased by two but by 1.5.
A.3 Examples for the vector space approach
Applying the extension using word embeddings on the two example Datasets 10 and 11 would use the words not occurring in both datasets. Creating from Dataset 10 might have the closest match with process from Dataset 11. Thus, the count for (VB, NN) would be increased, which clearly is a mismatch. The word an might have the lowest vector space distance to  $a$  from the other dataset. This accurate match would increase the count for (DT, DT). The remaining, so far unused, words from Dataset
DT	NN	VB	X	Σ
DT	5	0	0	0	5
NN	0	8	0	0	8
VB	0	2	2	0	4
X	0	0	0	6	6
Σ	5	10	2	6	23
Table 7: Contingency table derived from the additive combination of counts in Table 6.
10 have to be matched with their most similar counterparts from Dataset 11. For each pair of words, the count for the corresponding label-pair needs to be increased in the contingency table. While most vector representation matches between those two example datasets are inadequate, the quality of these matches is higher with larger datasets.
The application of the token-based approach using contextual embeddings on the two example Datasets 10 and 11 would work in the following way. All tokens in the two datasets are augmented with their corresponding contextual vector representations, thereby creating an associative array from a numeric vector to a label. For each word embedding in the first dataset, the vector representation with the closest distance from the other dataset is selected. Assuming the five matches are Creating-is, an-the, example-data, to-for and explain-is, the counts in a contingency table have to be increased for the label-pairs (VB, VB), (DT, DT), (NN, NN), (X, X) and (VB, VB).
A.4 Additional scores for the controlled environment experiments
The shared vocabulary values shown in Figure 3 exhibit a clear diagonal line of maximal shared vocabulary due to pairs of identical dataset samples. The remaining values are in accordance with the dataset sizes. For a chosen dataset, the shared vocabulary ratio increases with the size of the second dataset used in the comparison. Thus, there is no systematic difference between POS tagging and NER datasets nor a clear distinction between samples within the same original dataset and other datasets. Overall, the shared vocabulary is unsuitable to select datasets deemed similar.
Figure 4 shows the joint label entropy obtained from the same contingency tables as the NMI
Figure 3: Pairwise shared vocabulary ratio (Equation 8) between the twelve sampled datasets. The heat map encodes the values from 0.0 in black to 1.0 in white.
Figure 4: Pairwise joint label entropy values (Equation 2 and denominator of Equation 6) obtained on contingency tables filled with the vector space similarity approach using contextual BERT embeddings. The heat map encodes the values from min in black to max in white.
scores presented in Figure 2. While pairs of identical datasets exhibit a lower entropy relative to other pairs in the same row or column, there is no way to distinguish samples of the same original dataset from any other. The entropy values for NER-NER pairs are by far lower than any other pairs. This is reasonable as the "O" labels by far make up the majority of all labels in NER datasets. However, this does not help to find similar datasets in other cases, because there is no meaningful ordering of the entropy values when comparing any of the POS samples with all the other samples. In short, joint label entropy is not appropriate to find datasets deemed similar.
A.5 Neural network training procedure and hyperparameters
We train each model for at most 100 epochs with an early-stopping patience of 10 and a batch size of 256. The main and auxiliary training datasets are combined via interleaved batches from both datasets. Due to negligible effect, the dimensions of the character embeddings and hidden units are fixed at 32 resp. 64. 128 and 256 dimensions are tested for the word embeddings and the hidden units of the word GRU that can have either one or two layers. We use the Adam (Kingma and Ba, 2015) optimizer.
For POS tagging, the learning rate is fixed at 0.002. The best dropout value is chosen from the values 0, 0.25, 0.5, 0.75. Additional regularization via weight decay is selected from the values 0, 0.1, 0.01, 0.001.
For NER, the learning rate is set to 0.005 and weight decay uses a fixed value of 0.05. The range for dropout is narrowed to the values 0.3, 0.4, 0.5, 0.6. Each combination of hyperparameters is run with three random seeds to mitigate performance fluctuations due to the random initialization of the network weights. While the POS tagging experiments only used a Softmax classifier, we evaluate both Softmax and CRF classifiers for NER.