SlowGuess/ACL_2020_OCR huggingface dataset:
Overall, the  $NMI_{joint}$  scores presented in Figure 1 agree with the intuition which dataset sam
plies should be similar. For each row or column, the similarity values can be ordered descending by identical, same original dataset, same task, and other samples.
5.2 Experimental setup
Experiments to correlate dataset similarity and the network's multi-task learning performance will be performed a) using two neural network architectures with Softmax and conditional random field classifiers, b) for the tasks of POS tagging, NER, and AM, c) on multiple datasets per task. Table 2 shows the datasets used in the experiments. Similar to Yang et al. (2017), we sample new training datasets as subsets of the originals to show a larger influence of auxiliary data as there is no room for improvement for simple tasks on large training sets. For the auxiliary datasets, subsets of different sizes are sampled to allow a fair comparison of the performance effect. The standard development and test sets of the original datasets are used if available. Otherwise, random samples without overlap with any other subsampled dataset are used.
From the POS tagging datasets, a new training dataset of 25000 tokens is sampled for WSJ, BC, and EWT. From all POS tagging datasets, auxiliary datasets of increasing size are sampled containing 25, 50, 100, 250, 500,  $1000 \times 1000$  tokens limited by the size of the original dataset.
For NER, training sets of 50000 tokens are sampled from all datasets except GMB, SEC, and WNUT. Auxiliary datasets containing 50,100,  $250 \times 1000$  tokens are created for all datasets whenever possible.
For AM, we use the full PE and WD datasets for training and as auxiliary data. We sample auxiliary data from the IBM data equal in size to the others.
As the primary concern of the experiments is to enable significant differences in the neural network results with different auxiliary datasets, the network shares most of its parameters. In order to allow every training and auxiliary dataset combination to use their full potential, all relevant hyperparameters are tested for each pair of training and auxiliary dataset similar to Schulz et al. (2018).
The neural network architecture for the experiments uses hard parameter sharing with a bidirectional gated recurrent unit (GRU) (Cho et al., 2014), a simpler version of the long short-term memory (Hochreiter and Schmidhuber, 1997), that is commonly used in MTL sequence tagging works
See Figures 3 and 4 in Appendix A.4 for details.
ID	Dataset	Reference	Tokens	Tags	STL performance
PART-OF-SPEECH TAGGING DATASETS
BNC	British National Corpus	BNC Consortium (2007)	111 973 625	91	-
WSJ	Penn Treebank Wall Street Journal	Marcus et al. (1999)	1 286 980	45	86.35 ± 0.26
BC	Penn Treebank Brown Corpus	Marcus et al. (1999)	1 162 358	45	85.61 ± 0.35
EWT	UD English Web Treebank	Silveira et al. (2014)	254 854	17	88.35 ± 0.42
GSD	UD German GSD	McDonald et al. (2013)	297 836	17	-
NAMED-ENTITY RECOGNITION DATASETS
ONT	English OntoNotes Release 5.0	Weischedel et al. (2013)	2001 102	37	47.53 ± 0.83
CNLE	CoNLL'03 Shared Task (English)	Tjong Kim Sang and De Meulder (2003)	301 418	9	70.30 ± 2.50
CNLG	CoNLL'03 Shared Task (German)	Tjong Kim Sang and De Meulder (2003)	310 318	9	41.62 ± 0.27
EPG	Part of EUROPARL (German)	Faruqui and Padó (2010)	110 405	9	86.99 ± 0.42
GEN	GermEval 2014 NER Shared Task	Benikova et al. (2014)	591 005	24	26.97 ± 1.16
GMB	Groningen Meaning Bank 2.2.0	Bos et al. (2017)	1 354 149	17	-
SEC	SEC filings	Salinas Alvarado et al. (2015)	54 256	8	-
WIKI	Wikigold	Balasuriya et al. (2009)	39 152	8	67.19 ± 1.38
WNUT	W-NUT'17 Shared Task	Derczynski et al. (2017)	101 736	13	-
ARGUMENTATION MINING DATASETS
PE	Persuasive Essays (version 2)	Stab and Gurevych (2017)	148 182	11	53.71 ± 1.01
WD	Web Discourse	Habernal and Gurevych (2017)	84 817	12	24.58 ± 1.32
IBM	IBM Debater	Levy et al. (2018)	48 626 006	5	-
Table 2: Datasets used to sample new training or auxiliary datasets. The number of tags is a generic count, where e.g. B-PER and I-PER are considered to be different tags. STL performance (accuracy for POS, else macro F1 score) is not obtained on the full, but on the sampled training sets. STL scores are not shown for datasets only used as auxiliary data. Note that the IBM dataset contains many duplicate claims and near-duplicate sentences.
(see Section 2.1). Apart from self-learned word embeddings, character features based on another bidirectional GRU are included. Similar to Plank et al. (2016); Martínez Alonso and Plank (2017); Bjerva (2017); Ruder et al. (2019) we decided against pre-trained word embeddings in the network to avoid any influence on the comparison of STL and MTL performance. The last two, task-specific layers transform the GRU's hidden state to the task-specific labels and apply either a Softmax or conditional random field (CRF) (Lafferty et al., 2001) to predict the label. $^{6}$
Auxiliary data is only used for the same task, i.e. no POS tagging dataset is used as auxiliary training data for NER and vice versa. For POS tagging, 81 pairs of training and auxiliary datasets are tested with 64 hyperparameter combinations and three random seeds. In the case of NER, 117 pairs of training and auxiliary datasets are tested with two neural network models, 16 hyperparameter combinations, and three random seeds. In total, 26784 training runs have been performed.
We compute the similarities for pairs of training and auxiliary datasets in three ways. The text overlap approach is used with and without word embeddings. For the latter, 300-dimensional fastText
embeddings<sup>7</sup> with sub-word information are used that consist of 2 million word vectors trained on the Common Crawl (Mikolov et al., 2018). We evaluate the additive and multiplicative ways with multiple weighting schemes to combine the label counts and calculate various similarity measures from the resulting contingency table. The "BERT-Base Multilingual Cased" model (Devlin et al., 2019) is used for the third, token-based approach.
5.3 Results and analysis
In Figure 2, the difference in accuracy over STL is plotted against the UUE  $NMI_{joint}$  similarity measure using BERT embeddings. Overall, the data points are scattered from the bottom left to the top right. There are no cases of low similarity coinciding with high accuracy increase. The data points with auxiliary data from the German GSD dataset are clustered close to the bottom left, i.e. low similarity and almost no accuracy gain. This concurs with the intuition that using a German auxiliary dataset for an English training dataset should not lead to a significant performance increase. The data points with auxiliary data from the same original dataset as the training set are clustered to the top right, i.e. have the highest similarity and performance increase as expected. The scatter plots for
$^{6}$ Training procedure and hyperparameters are described in more detail in Appendix A.5
$^{7}$  crawl300d2Msubword.zip from fasttext.cc
other sizes of auxiliary data and methods, e.g. computing  $NMI_{max}$  on the contingency table from the text overlap approach, look similar.
Figure 2: Plot comparing the POS tagging difference in accuracy between STL and MTL (auxiliary size 250000 tokens) with the UUE  $NMI_{joint}$  similarity obtained using BERT embeddings for each token
To quantify the various similarity computation methods, we correlate the change in accuracy with the similarity value. Table 3 shows the median and mean correlation of similarity with change in accuracy for the best ten methods averaged over groups of identically-sized auxiliary datasets. As a baseline, the correlation with the ratio of shared vocabulary is included. We only show the results for  $NMI_{joint}$  as the correlation was equal to or better than  $NMI_{max}$  in most cases. The correlation between the similarity and change in accuracy is strong according to both Kendall's rank correlation and Pearson's linear correlation coefficients,
which is in line with the plot shown in Figure 2. Since the  $p$ -values for the similarity methods are well below 0.005, it is very unlikely that similarity and accuracy are not correlated. The strongest correlation, according to Kendall's  $\tau$ , is achieved with the harmonic mean of shared vocabulary and multiplicative text overlap. According to Pearson's  $\rho$ , the highest linear correlation is achieved with the UUE (Equation 9) vector space method, which is depicted in Figure 2. The correlation coefficients of the text overlap approach are consistently higher than the shared vocabulary baseline since the baseline is oblivious to the labels.
For NER, the results are shown in Table 4. In comparison to the POS tagging results, methods using embeddings perform better than those without. The strongest Kendall and Pearson correlations are achieved by the vector space approach computing the joint NMI on a contingency table filled from forward BERT embeddings. While a linear correlation on the POS tagging results was deemed reasonable based on a data analysis, the Pearson correlation values for NER might be prone to outlier effects and are therefore only included for completeness.
For AM, no quantitative analysis could be performed due to a limited number of samples. With MTL, the performance on PE increased to 54.26 when using WD as auxiliary data, while IBM reduced it to 51.37. WD performance is slightly reduced by PE as auxiliary data to 21.72, but reduced to 9.42 by IBM. While we saw no correlation with the text overlap similarities, the forward vector space measure matches the MTL score change
Primary method	Combination	Count method	Embedding	τ	Kendall's τ	ρ	Pearson's ρ
text overlap & SV	TO	multiplicative	-	0.73	0.71 ± 0.05	0.80	0.79 ± 0.07
text overlap & SV	TO	additive	-	0.72	0.72 ± 0.10	0.78	0.79 ± 0.04
text overlap	-	multiplicative	fastText	0.70	0.69 ± 0.08	0.83	0.82 ± 0.07
vector space	UUE	-	BERT	0.70	0.69 ± 0.12	0.84	0.84 ± 0.06
vector space	-	-	BERT	0.69	0.65 ± 0.09	0.83	0.82 ± 0.06
text overlap	-	multiplicative	-	0.68	0.64 ± 0.12	0.73	0.74 ± 0.08
text overlap	UUE	additive	-	0.67	0.66 ± 0.12	0.75	0.77 ± 0.06
text overlap	-	additive	-	0.67	0.65 ± 0.11	0.74	0.76 ± 0.06
text overlap	-	additive	-	0.66	0.64 ± 0.12	0.68	0.69 ± 0.08
text overlap	UUE	multiplicative	fastText	0.65	0.65 ± 0.11	0.83	0.83 ± 0.04
shared vocabulary	-	-	-	0.63	0.60 ± 0.14	0.77	0.75 ± 0.07
Table 3: Correlation between various  $NMI_{joint}$  similarity measures and the change in POS tagging accuracy using MTL. The entries show the median and mean of Kendall's and Pearson's correlation coefficients sorted descendingly by  $\tilde{\tau}$ . The average  $p$ -values for all methods (except the shared vocabulary baseline) are below 0.005.
Primary method	Combination	Count method	Embedding	τ	Kendall's τ	ρ	Pearson's ρ
vector space	-	-	BERT	0.65	0.62 ± 0.06	0.95	0.92 ± 0.05
vector space	UUE	-	BERT	0.59	0.55 ± 0.11	0.89	0.89 ± 0.05
text overlap	-	multiplicative	fastText	0.57	0.54 ± 0.09	0.91	0.88 ± 0.07
text overlap	-	additive	fastText	0.57	0.54 ± 0.09	0.87	0.86 ± 0.05
text overlap	UUE	multiplicative	fastText	0.52	0.50 ± 0.13	0.80	0.83 ± 0.06
text overlap & SV	TO	additive	-	0.51	0.50 ± 0.13	0.81	0.79 ± 0.04
text overlap & SV	TO	multiplicative	-	0.51	0.50 ± 0.13	0.80	0.79 ± 0.06
text overlap	UUE	additive	fastText	0.49	0.48 ± 0.08	0.83	0.84 ± 0.04
text overlap	-	multiplicative	-	0.47	0.44 ± 0.11	0.83	0.82 ± 0.08
text overlap	-	additive	-	0.42	0.41 ± 0.07	0.82	0.80 ± 0.04
shared vocabulary	-	-	-	0.48	0.49 ± 0.13	0.75	0.73 ± 0.05
Table 4: Correlation between  $NMI_{joint}$  various similarity measures and the change in NER F1 score using MTL. The entries show the median and mean of Kendall's and Pearson's correlation coefficients sorted descendingly by  $\tilde{\tau}$ . The average  $p$ -values for all methods (except the shared vocabulary baseline) are below 0.001. The change in F1 score was highly affected by random initialization, so the correlation scores must be used with caution.
when comparing averaged span embeddings: The  $NMI_{joint}$  similarity of PE-IBM is 0.09, and PE-WD is measured 0.26 whereas WD-PE has a similarity score of 0.06 and WD-IBM is scored 0.04. Thus, our similarity measure identifies the most promising auxiliary dataset also in this case.
Overall, there is a strong correlation between MTL scores and dataset similarity computed by our proposed methods. In the case of POS tagging, the correlation is impressive — it is visible in the scatter plot and accompanied by high-confidence correlation coefficients. The results for NER are less clear but still indicate that similarity and test set performance are correlated.
We can recommend the text overlap approach combined with the shared vocabulary for syntactic tasks with single-token labels. It performed the best in our POS tagging evaluation and is computed in less than a second. Both additive and multiplicative count combination methods worked equally well in our tests. For more complex tasks such as NER or AM and in case labels span multiple tokens, we suggest using the approach based on the forward vector space similarity. It performed the best in our NER evaluation. Further, it was the only method to work reasonably well with the AM datasets because spans of multiple tokens could be compared by combining the embeddings of all contained tokens. In all cases, we recommend using the mutual information normalized by the joint entropy  $NMI_{joint}$  as the actual similarity measure because it was either equal to or better than the other variants.
The similarity measures allow distinguishing good from bad candidates for usage as auxiliary data. This is an immensely valuable information as the number of expensive neural network training runs can be reduced to a fraction while still finding the best auxiliary dataset(s) to increase performance on the main task. In contrast to previous methods, our measures do not require the label sets to be the same and do not require automatic tagging. The experiments show that similarity measures allow ordering the effects of auxiliary datasets by direction and intensity for an individual training dataset. Our experimental findings are also supported from a theoretical point of view. The developed methods working on both words and their labels have a substantial advantage over approaches that are based only on words or the label distributions. The quick similarity calculation can improve the main task performance when better datasets are used as auxiliary data that would never have made it through the otherwise purely manual preselection process.
In future work, apart from improving the similarity measures, it could be examined to predict MTL scores or estimate the right amount of auxiliary data or shared parameters in the neural network.
We would like to thank all anonymous reviewers for their valuable feedback. This work was partially funded by the Cluster of Excellence CLICCS (EXC 2037), Universität Hamburg, funded through the German Research Foundation (DFG).
Isabelle Augenstein, Sebastian Ruder, and Anders Søgaard. 2018. Multi-task learning of pairwise sequence classification tasks over disparate label spaces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1896-1906, New Orleans, Louisiana. Association for Computational Linguistics.
Isabelle Augenstein and Anders Søgaard. 2017. Multi-task learning of keyphrase boundary classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 341-346, Vancouver, Canada. Association for Computational Linguistics.
Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. 2009. Named entity recognition in Wikipedia. In Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources (People's Web), pages 10-18, Suntec, Singapore. Association for Computational Linguistics.
Jonathan Baxter. 2000. A model of inductive bias learning. Journal of Artificial Intelligence Research (JAIR), 12(1):149-198.
Shai Ben-David and Reba Schuller. 2003. Exploiting task relatedness for multiple task learning. In Learning Theory and Kernel Machines, pages 567-580, Berlin, Heidelberg. Springer Berlin Heidelberg.
Darina Benikova, Chris Biemann, and Marc Reznicek. 2014. NoSta-d named entity annotation for German: Guidelines and dataset. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 2524-2531, Reykjavik, Iceland. European Language Resources Association (ELRA).
Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 164-169, Valencia, Spain. Association for Computational Linguistics.
Johannes Bjerva. 2017. Will my auxiliary tagging task help? estimating auxiliary tasks effectivity in multi-task learning. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 216-220, Gothenburg, Sweden. Association for Computational Linguistics.
Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic tagging with deep residual networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3531-3541, Osaka, Japan. The COLING 2016 Organizing Committee.
BNC Consortium. 2007. The British National Corpus, version 3 (BNC XML Edition).
Johan Bos, Valerio Basile, Kilian Evang, Noortje Venhuizen, and Johannes Bjerva. 2017. The Groningen Meaning Bank. In Nancy Ide and James Pustejovsky, editors, Handbook of Linguistic Annotation, volume 2, pages 463-496. Springer.
Rich Caruana. 1997. Multitask learning. Machine Learning, 28(1):41-75.
Soravit Changpinyo, Hexiang Hu, and Fei Sha. 2018. Multi-task learning for sequence tagging: An empirical study. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2965-2977, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, New York, New York, USA.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ábel Elekes, Martin Schäler, and Klemens Böhm. 2017. On the various semantics of similarity in word embedding models. In Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries, JCDL '17, pages 139-148, Toronto, Ontario, Canada. IEEE Press.
Manaal Faruqui and Sebastian Padó. 2010. Training and evaluating a German named entity recognizer with semantic generalization. In Semantic Approaches in Natural Language Processing: Proceedings of the 10th Conference on Natural Language Processing, KONVENS 2010, pages 129-133, Saarbrücken, Germany.
Ivan Habernal and Iryna Gurevych. 2017. Argumentation mining in user-generated web discourse. Computational Linguistics, 43(1):125-179.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Suzana Ilic, Edison Marrese-Taylor, Jorge Balazs, and Yutaka Matsuo. 2018. Deep contextualized word representations for detecting sarcasm and irony. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 2-7, Brussels, Belgium. Association for Computational Linguistics.
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. 2017. One model to learn them all. arXiv:1706.05137.
Joo-Kyung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017. Cross-lingual transfer learning for POS tagging without cross-lingual resources. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2832–2838, Copenhagen, Denmark. Association for Computational Linguistics.
Young-Bum Kim, Karl Stratos, Ruhi Sarikaya, and Minwoo Jeong. 2015. New transfer learning techniques for disparate label sets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 473-482, Beijing, China. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, San Diego, California, USA.
T. O. Kvalseth. 1987. Entropy and correlation: Some comments. IEEE Transactions on Systems, Man, and Cybernetics, 17(3):517-519.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, pages 282-289, San Francisco, California, USA. Morgan Kaufmann Publishers Inc.
Ran Levy, Ben Bogin, Shai Gretz, Ranit Aharonov, and Noam Slonim. 2018. Towards an argumentative content search engine using weak supervision. In Proceedings of the 27th International Conference on