SlowGuess/NAACL_2022_OCR huggingface dataset:
6 representative languages (the trend for other languages is similar). We find that for parsing, the size of our method's chart is much smaller than  $1\%$  of the size of PPTX chart for all target languages. This finding shows that our method's charts are much more compact than those of PPTX. Thus, it may explain the improvement of our method over PPTX because smaller charts may be more likely to concentrate on trees that have many correct arcs, making it easier for the model to learn correctly (we explore this further in Section 4.2). For POS tagging, we observe the same trend where our method's charts are smaller, but to a lesser extent, presumably because the typical output space of tagging is several orders of magnitude smaller than that of parsing. Occasionally, our method's chart is larger than that of PPTX, although our method outperforms PPTX substantially (French and Spanish). We speculate that this is because most of the source taggers are very confident but on different substructures, so only a handful of substructures are selected by PPTX after applying the threshold in Eq. (2), making the chart small. Meanwhile, the logarithmic opinion pool is less confident as it corresponds to the (geometric) mean of the distributions, so more substructures are selected, making the chart larger.
4.2 Chart Quality Analysis
Continuing the previous analysis, we check if the smaller charts of our method indeed concentrate more on the correct structures than those of PPTX. To measure this, we define the notion of precision and recall of the chart  $\tilde{\mathcal{V}}(x)$ . We define precision as the fraction of correct substructures in  $\tilde{\mathcal{V}}(x)$  and recall as the fraction of gold substructures that occur in any structure in  $\tilde{\mathcal{V}}(x)$ . Formally,
$$
\mathrm {P} (\tilde {\mathcal {Y}} (\boldsymbol {x})) = \frac {\sum_ {(\boldsymbol {x} , \boldsymbol {y} ^ {*})} \sum_ {\boldsymbol {y} \in \tilde {\mathcal {Y}} (\boldsymbol {x})} | A (\boldsymbol {y}) \cap A (\boldsymbol {y} ^ {*}) |}{\sum_ {\boldsymbol {x}} \sum_ {\boldsymbol {y} \in \tilde {\mathcal {Y}} (\boldsymbol {x})} | A (\boldsymbol {y}) |}
$$
$$
\operatorname {R} (\tilde {\mathcal {Y}} (\boldsymbol {x})) = \frac {\sum_ {(\boldsymbol {x} , \boldsymbol {y} ^ {*})} \sum_ {a \in A (\boldsymbol {y} ^ {*})} \operatorname {I} (a , \tilde {\mathcal {Y}} (\boldsymbol {x}))}{\sum_ {\boldsymbol {y} ^ {*}} | A (\boldsymbol {y} ^ {*}) |}
$$
$$
\operatorname {I} (a, \tilde {\mathcal {Y}} (\boldsymbol {x})) = \left\{ \begin{array}{l l} 1 & \text {i f} \boldsymbol {y} \in \tilde {\mathcal {Y}} (\boldsymbol {x}) \text {s . t .} a \in A (\boldsymbol {y}) \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
6Except for Turkish, where this number is  $3\%$  which is still very small.
Target Language	Parsing	Tagging
PPTX	Ours	Δ	PPTX	Ours	Δ
P	R	P	R	P	R	Acc	P	R	P	R	P	R	Acc
fa	10	90	17	95	7	5	6.9	21	80	26	75	5	4	3.7
ko	0	65	2	77	1	13	-1.0	8	50	4	45	-5	-5	0.2
hr	10	96	20	98	10	3	2.1	14	77	16	77	2	0	10.1
it	10	99	25	100	15	1	1.2	20	91	24	93	4	2	22.9
es	11	96	20	98	10	1	1.6	18	82	16	87	-1	4	15.7
sv	13	97	21	98	8	1	0.2	21	84	28	81	8	-3	0.4
Table 3: Precision (P) and recall (R) of charts produced by PPTX and our method in dependency parsing and POS tagging. Numbers are rounded to the nearest integer. Column  $\Delta$  is the difference between our method and PPTX (positive means our method is higher).  $\Delta$  over the accuracy results for both tasks are included for completeness, and correspond to the bar height difference of the two methods in Fig. 3.
and  $y^{*}$  denotes the gold structure for input  $x$ . A good chart must have high precision and recall. In particular, if  $\tilde{\mathcal{V}}(x)$  is a singleton containing the gold structure, then both precision and recall will be  $100\%$ .
Table 3 reports the precision and recall of the charts produced by PPTX and our method for both tasks, as well as the performance differences, for the same 6 languages as before (the trend for other languages is similar). We observe that with our method in parsing, both precision and recall consistently improve over PPTX, suggesting that the charts indeed contain more correct arcs. However, higher precision and recall do not guarantee performance improvement, as shown by Korean where both precision and recall improve with our method but its performance is lower than PPTX.7 We suspect that this is caused by the unusually low precision even with our method, indicating that the chart is very noisy. For POS tagging, the result is less obvious, but we find that generally our method improves chart precision, but often sacrificing chart recall. For Spanish, precision decreases with our method, and only recall improves.8 An interesting case is again Korean where both precision and recall worsen, probably because of very poor source taggers performance on the language. Overall, our method generally improves the chart quality in terms of either precision or recall, but to a lesser extent, which again may be attributed to the smaller output space compared with parsing.
4.3 Effect of Opinion Pool Distance to True Distribution
We explore whether there is a relationship between (a) how distant the opinion pool is to the true distribution over substructures and (b) the performance improvement of our method against majority voting. Intuitively, the closer the opinion pool is to the true distribution, the higher its absolute performance would be. However, it is unclear whether this translates into an advantage over majority voting. This is important because if such relationship exists, then it may be worthwhile spending some effort on optimising the opinion pool. To this end, we measure the distance between the true distribution and the opinion pool by computing the Kullback-Leibler divergence (KL)
$$
\operatorname {K L} (\hat {p} \mid \bar {p}) = \frac {1}{n (\mathcal {D})} \sum_ {\boldsymbol {x} \in \mathcal {D}} \sum_ {j = 1} ^ {| \boldsymbol {x} |} \operatorname {K L} (\hat {p} _ {j} \mid \bar {p} _ {j}) \tag {5}
$$
where  $n(\mathcal{D})$  is the total number of tokens of all input sentences in  $\mathcal{D}$ ,  $\hat{p}_j(c \mid x)$  is the (empirical) true distribution of substructures in  $C(x, j)$ , and  $\bar{p}_j(c \mid x)$  is the logarithmic opinion pool distribution defined in Eq. (4). Note that  $\hat{p}_j(c \mid x)$  is a one-hot distribution so  $\mathrm{KL}(\hat{p}_j \mid \bar{p}_j)$  reduces to the negative log likelihood of the labelled data under the opinion pool. We compute the KL divergence on the training set of both parsing and tagging and display the regression plots in Fig. 4. We see a medium correlation between opinion pool distance and performance gain against majority voting, with  $r = -0.45$  for both parsing and tagging ( $p$ -value is 0.06 for both). However, there is substantial variance, especially in the right half figure of parsing, caused by the lack of languages in that region of the plot. Nonetheless, the plots suggest that there
The only other language where this happens is Hindi.
The only other language where this happens is French.
(a) Dependency parsing
Figure 4: Relationship between  $\mathrm{KL}(\hat{p} \mid \bar{p})$  and the accuracy difference of our method and MV, where  $\hat{p}$  and  $\bar{p}$  denote the empirical true distribution and the opinion pool distribution respectively. Shaded area is  $95\%$  confidence interval computed via bootstrapping.
(b) POS tagging
is indeed a positive relationship between how close the opinion pool is to the true distribution and the performance gain of our method compared with majority voting.
There are ways to obtain an opinion pool that is closer to the true distribution. One way is to leverage a small amount of labelled data in the target language to estimate the weight factors  $\alpha_{k}$ , which can be done by optimising Eq. (5). This option is suitable if such labelled data is available or can be obtained cheaply. If we have the freedom to choose the source languages, another method is to select them carefully so they are both reasonably close to the target language and also diverse. This is because Eq. (5) can be expressed as  $E - D$ , where  $E$  denotes how distant the source models' output distributions are to the target's true distribution (error) and  $D$  denotes how distant the output distributions are to each other (diversity) (Heskes, 1998). Having the source languages reasonably close to the target language and also diverse means reducing  $E$  and increasing  $D$  respectively, moving the opinion pool closer to the true distribution. That said, when the source languages are close to the target language, the source models may already be good for direct transfer so our method may not give meaningful improvement over majority voting.
4.4 Learning the Opinion Pool Weight Factors
Motivated by the previous findings, we deviate from our unsupervised setup by learning the weight factors  $\alpha_{k}$  using a tiny amount of labelled target data. Concretely, we randomly sample 50 sentences from the training set of each target language and learn  $\alpha_{k}$  that minimises Eq. (5) for all source model  $k$ . We then use the learned weights to obtain the opinion pool as defined in Eq. (4) (see Ap
Parsing	Tagging
MV	56.3	65.4
Uniform αk	59.0	69.3
Learned αk	59.4	70.0
Table 4: Parsing and tagging performance of MV and our method with uniform and learned weight factors  $\alpha_{k}$  for the logarithmic opinion pool, averaged over 18 languages. Full results are reported in Table 11.
pendix F for further details). Table 4 shows the results on parsing and tagging, averaged over the target languages. We observe that by using the learned weight factors, our method slightly improves over the version using uniform weights, suggesting that our method can readily leverage labelled target data if it is available. On the other hand, the fact that the improvement is only modest also reaffirms that uniform weighting is a strong baseline.
A straightforward method of multi-source transfer is training a model on the concatenation of datasets from the source languages. This approach was used by McDonald et al. (2011) for dependency parsing and yields a substantial gain compared with single-source transfer. More recent work by Guo et al. (2016) proposed to learn multilingual representations from the concatenation of source language data and use them to train a neural dependency parser. Another method is language adversarial training, used by Chen et al. (2019) for various NLP tasks including named-entity recognition, which is often cast as structured prediction. Despite their success, multi-source unsupervised cross-lingual transfer methods typically require the source language data, which is not always feasible.
There are recent methods suitable in this source-free setup. Rahimi et al. (2019) proposed a method based on truth inference to model label confusion in multi-source transfer of named-entity recognisers. Wu et al. (2020) used teacher-student learning for named-entity recognition. A closely related work is by Hu et al. (2021) who argued that a small amount of labelled data in the target language is cheap to obtain and proposed an attention-based method to weight the source models leveraging 50 labelled target sentences. Our work is different as we do not require any labelled data and evaluate on 3 times more languages than they did. In addition, their model is based on mBERT (Devlin et al., 2019), which benefits from larger data from
multiple languages during pretraining. In many application scenarios, BERT-based models are too costly, especially when criteria other than accuracy matter (Nityasya et al., 2020).
Our work builds upon the work of Kurniawan et al. (2021) who proposed a method based on self-training for unsupervised cross-lingual dependency parsing. In this work, we generalise their method to structured prediction problems and propose a modification to improve the quality of the distant supervision.
In this paper, we (1) generalise previous methods for cross-lingual unsupervised transfer without source data to structured prediction problems and (2) propose a new aggregation technique which can better handle mixed-quality input distributions. Experiments across two structured prediction tasks and 18 languages show that, unlike previous work, our method generally outperforms a strong multisource transfer baseline. Our analyses suggest that our method produces distant supervision of better quality than that of the previous methods. Our work potentially generalises beyond language transfer to (a) structured prediction tasks beyond NLP and (b) transfer across other types of domains (e.g., genres), a direction we aim to explore in future work. We are also interested in investigating in future work whether our method helps transfer with recent multilingual pretrained models.
Wasi Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, and Nanyun Peng. 2019. On Difficulties of Cross-Linguual Transfer with Order Differences: A Case Study on Dependency Parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2440–2452.
Alena Böhmová, Jan Hajič, Eva Hajičová, and Barbora Hladká. 2003. The Prague Dependency Treebank. In Anne Abeillé, editor, Treebanks: Building and Using Parsing Corpora, pages 103-127.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Thorsten Brants, Wojciech Skut, and Hans Uszkoreit. 2003. Syntactic Annotation of A German Newspaper
Corpus. In Anne Abeille, editor, Treebanks: Building and Using Parsing Corpora, pages 73-87.
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-Source Cross-Lingual Model Transfer: Learning What to Share. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3098-3112.
Yoeng-Jin Chu and Tseng-Hong Liu. 1965. On the shortest arborescence of a directed graph. Scientia Sinica, 14:1396-1400.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Timothy Dozat and Christopher D Manning. 2017. Deep Biaffine Attention for Neural Dependency Parsing. In International Conference on Learning Representations, page 8.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015. Cross-lingual Transfer for Unsupervised Dependency Parsing without Parallel Data. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 113–122.
Jack Edmonds. 1967. Optimum branchings. Journal of Research of the national Bureau of Standards B, 71(4):233-240.
Klaus Greff, Aaron Klein, Martin Chovanec, Frank Hutter, and Jürgen Schmidhuber. 2017. The Sacred Infrastructure for Computational Research. In Proceedings of the 16th Python in Science Conference, pages 49-56.
Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A Representation Learning Framework for Multi-Source Transfer Parsing. In Thirtieth AAAI Conference on Artificial Intelligence.
Junxian He, Zhisong Zhang, Taylor Berg-Kirkpatrick, and Graham Neubig. 2019. Cross-Lingual Syntactic Transfer through Unsupervised Adaptation of Invertible Projections. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3211-3223.
Tom Heskes. 1998. Selecting weighting factors in logarithmic opinion pools. In Advances in Neural Information Processing Systems, volume 10.
Zechuan Hu, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Multi-View Cross-Linguual Structured Prediction with Minimum Supervision. In Proceedings of the 59th Annual Meeting of the Association for Computational
Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2661-2674.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations.
Kemal Kurniawan, Lea Frermann, Philip Schulz, and Trevor Cohn. 2021. PPT: Parsimonious Parser Transfer for Unsupervised Cross-Linguual Adaptation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2907-2918.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective Self-Training for Parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152-159.
Ryan McDonald, Fernando Pereira, Kiril Rbarov, and Jan Hajic. 2005. Non-Projective Dependency Parsing using Spanning Tree Algorithms. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 523-530.
Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-Source Transfer of Delexicalized Dependency Parsers. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 62–72.
Made Nindyatama Nityasya, Haryo Akbario T Wibowo, Radityo Eko Prasojo, and Alham Fikri Aji. 2020. No Budget? Don't Flex! Cost Consideration when Planning to Adopt NLP for Your Business. arXiv:2012.08958 [cs].
Joakim Nivre, Mitchell Abrams, Željko Agić, and et al. 2018. Universal Dependencies 2.2. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Sinno Jialin Pan and Qiang Yang. 2010. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-1359.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors,
Advances in Neural Information Processing Systems 32, pages 8024-8035.
Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal Dependency Parsing from Scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160-170.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively Multilingual Transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164.
Alexander Rush. 2020. Torch-Struct: Deep Structured Prediction Library. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 335–342.
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In International Conference on Learning Representations.
Qianhui Wu, Zijia Lin, Börje Karlsson, Jian-Guang LOU, and Biqing Huang. 2020. Single-/Multi-Source Cross-Linguual NER via Teacher-Student Learning on Unlabeled Data in Target Language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6505-6514.
A Evaluation Languages
Table 5 lists the languages we use in our evaluation, along with their family, subgroup (if the language is Indo-European), selected treebanks in Universal Dependencies v2.2, and the corresponding licence. The treebank selection follows Kurniawan et al. (2021) to enable a fair comparison. Unless stated otherwise, the licence is Creative Commons (CC).
Language	Code	Family	Treebanks	Licence
Distant
Persian	fa	IE.Iranian	Seraji	BY-SA 4.0
Arabic	ar	Afro-Asiatic	PADT	BY-NC-SA 3.0
Indonesian	id	Austronesian	GSD	BY-NC-SA 3.0†
Korean	ko	Koreanic	GSD	BY-NC-SA 3.0†
Kaist	BY-SA 4.0
Turkish	tr	Turkic	IMST	BY-NC-SA 3.0
Hindi	hi	IE.Indic	HDTB	BY-NC-SA 4.0
Croatian	hr	IE.Slavic	SET	BY-SA 4.0
Hebrew	he	Afro-Asiatic	HTB	BY-NC-SA 4.0
Nearby
Bulgarian	bg	IE.Slavic	BTB	BY-NC-SA 4.0
Italian	it	IE.Romance	ISDT	BY-NC-SA 3.0
Portuguese	pt	IE.Romance	GSD	BY-NC-SA 3.0†
Bosque	BY-SA 4.0
French	fr	IE.Romance	GSD	BY-NC-SA 3.0†
Spanish	es	IE.Romance	GSD	BY-NC-SA 3.0†
AnCora	GPL 3.0
Norwegian	no	IE.Germanic	Bokmaal	BY-SA 4.0
Nynorsk	BY-SA 4.0
Danish	da	IE.Germanic	DDT	BY-SA 4.0
Swedish	sv	IE.Germanic	Talbanken	BY-SA 4.0
Dutch	nl	IE.Germanic	Alpino	BY-SA 4.0
LassySmall	BY-NC-SA 4.0
German	de	IE.Germanic	GSD	BY-NC-SA 3.0†
B Source Models Performance
To train the source models, we tune the hyperparameters on English and use the values for training on the other source languages. Table 6 reports the performance of our source parsers and taggers. We also report the performance numbers of previous work, copied from their respective papers, to serve as reference. Generally, the source models achieve in-language performance comparable to previous work (e.g., Ahmad et al., 2019) with the exception of the Arabic parser whose accuracy is noticeably lower, possibly caused by the model
architecture optimised for transfer rather than in-language evaluation. However, we argue that the lower performance reflects a realistic application scenario where some of the source models are expected to be poor.
Table 5: List of languages in our evaluation, grouped into distant and nearby languages based on their distance to English (He et al., 2019). IE stands for Indo-European. †: licence is the United States version.
en	ar	es	fr	de
Parsing	86.9	76.9	90.0	89.1	82.1
Tagging	94.5	95.4	96.5	96.5	92.1
Previous work (reference only)
LSTM parser	88.3	81.8	90.8	89.1	83.7
Stanza tagger*	95.4	94.9	96.7	97.3	94.1
Table 6: Parsing and tagging accuracy of the source models. We copy numbers of the LSTM parser (Ahmad et al., 2019) and Stanza tagger (Qi et al., 2018) from their respective papers to serve as reference only. * indicates that the numbers are not directly comparable to ours because of the difference in the evaluation setup.
C Additional Baseline Details
Knowledge Distillation We use a similar method to the soft-KD baseline used by Hu et al. (2021), which was based on the teacher-student learning method of Wu et al. (2020). Let  $p_{\mathrm{head}}^{(k)}(h_j = i \mid x)$  denote the probability of  $x_j$  having  $x_i$  as head under source parser  $k$ . Similarly, let  $p_{\mathrm{label}}^{(k)}(l_{ij} = r \mid x)$  denote the probability of the arc between head  $x_i$  and dependent  $x_j$  having label  $r$  under source parser  $k$ . These distributions are obtained from the output of the corresponding biaffine layer that is then passed through a softmax layer. We average these distributions over the source parsers to give
$$
\bar {p} _ {\text {h e a d}} (h _ {j} = i \mid \boldsymbol {x}) = \frac {1}{K} \sum_ {k} p _ {\text {h e a d}} ^ {(k)} (h _ {j} = i \mid \boldsymbol {x}),
$$
$$
\bar {p} _ {\mathrm {l a b e l}} (l _ {i j} = r \mid \boldsymbol {x}) = \frac {1}{K} \sum_ {k} p _ {\mathrm {l a b e l}} ^ {(k)} (l _ {i j} = r \mid \boldsymbol {x})
$$
where  $K$  is the number of source parsers. The training objective of this KD baseline is then