SlowGuess/NAACL_2022_OCR huggingface dataset:
Kemal Kurniawan $^{1,2*}$  Lea Frermann $^{1}$  Philip Schulz $^{2\dagger}$  Trevor Cohn $^{1}$
$^{1}$ School of Computing and Information Systems, University of Melbourne
2Amazon Research
kemal.kurniawan@student.unimelb.edu.au
{lea.frermann, tcohn} @unimelb.edu.au
phschulz@amazon.com
Providing technologies to communities or domains where training data is scarce or protected e.g., for privacy reasons, is becoming increasingly important. To that end, we generalise methods for unsupervised transfer from multiple input models for structured prediction. We show that the means of aggregating over the input models is critical, and that multiplying marginal probabilities of substructures to obtain high-probability structures for distant supervision is substantially better than taking the union of such structures over the input models, as done in prior work. Testing on 18 languages, we demonstrate that the method works in a cross-lingual setting, considering both dependency parsing and part-of-speech structured prediction problems. Our analyses show that the proposed method produces less noisy labels for the distant supervision.<sup>1</sup>
Recent successes of NLP systems have been enabled by supervised learning algorithms requiring a large amount of labelled data. Creating such data can be costly for structured prediction tasks such as dependency parsing (Böhmová et al., 2003; Brants et al., 2003). Transfer learning (Pan and Yang, 2010) is a promising solution to this problem. In this work, we focus on a case of transfer learning, namely cross-lingual learning. We consider the setup where the target language is low-resource having only unlabelled data, commonly referred to as unsupervised cross-lingual transfer. This is an important problem because most world's languages are low-resource (Joshi et al., 2020). Successful transfer from high-resource languages enables language technologies development for these low-resource languages.
One recent method for unsupervised crosslingual transfer is PPTX (Kurniawan et al., 2021). Developed for dependency parsing, it transfers from multiple source languages, which has been shown to be superior to transferring from just a single language (McDonald et al., 2011; Duong et al., 2015; Rahimi et al., 2019, inter alia). PPTX computes the union of high-probability trees from all source parsers and uses the result as supervision to train the target language parser. One advantage is that, in addition to not requiring labelled data in the target language, it does not require any data in the source languages either, which is useful if such data is private. All it needs is access to multiple, trained source parsers. Despite its benefits, PPTX has only been applied to dependency parsing, although in principle it should be extensible to other structured prediction problems. More concerningly, we show in this work that PPTX generally underperforms compared to a majority voting baseline.
In this paper, we generalise and improve PPTX for structured prediction problems. As with PPTX, this generalisation casts the unsupervised transfer problem as a supervised learning task with distant supervision, where the label of each sample in the target language is based on the structures predicted by an ensemble of source models. Moreover, we propose the use of logarithmic opinion pooling (Heskes, 1998) to improve performance (see Fig. 1). Unlike PPTX that performs simple union, the pooling considers the output probabilities in aggregating the source model outputs to obtain the structures used for distant supervision. We test our method on 18 languages from 5 language families and on two structured prediction tasks in NLP: dependency parsing and POS tagging. We find that our method generally outperforms both PPTX and the majority voting baseline, with absolute accuracy gains of up to  $7\%$  on parsing and  $20\%$  on tagging. Our analysis shows that the use of logarithmic opinion pooling results in fewer predicted
*Work done prior to joining Amazon.
†Work done outside Amazon.
1https://github.com/kmkurn/uxtspwsd
Human Language Technologies, pages 2041 - 2054
Figure 1: Illustration of our method for an input sentenceuya makan nasi ("I eat rice").  $①$  A set of structured prediction models as inputs.  $②$  The models compute marginal probability distributions over substructures for each token  $x_{j}$ $③$  Logarithmic opinion pool of the distributions is computed.  $④$  Substructures are filtered based on some threshold.  $⑤$  High-probability substructures are obtained.  $⑥$  High-probability structures are obtained from the substructures as distant supervision.
structures that are also more concentrated on the correct ones.
In summary, our contributions in this paper are:
1. developing a generic unsupervised multisource transfer method for structured prediction problems;
2. leveraging logarithmic opinion pooling to take into account source model probabilities in the aggregation to produce the labels for distant supervision; and
3. outperforming previous work in dependency parsing and part-of-speech tagging, especially in the context of a stronger, multi-source transfer baseline.
2 Unsupervised Transfer as Supervised Learning
Suppose we want to create a model for a low-resource language that has only unlabelled data, but we only have access to a set of models trained on other languages. This is an instance of cross-lingual transfer learning. We cast this problem as a (distantly) supervised learning task with the training objective
$$
\ell (\boldsymbol {\theta}) = - \sum_ {\boldsymbol {x} \in \mathcal {D}} \log \sum_ {\boldsymbol {y} \in \tilde {\mathcal {Y}} (\boldsymbol {x})} p (\boldsymbol {y} \mid \boldsymbol {x}; \boldsymbol {\theta}) \tag {1}
$$
where  $\theta$  is the target model parameters,  $\mathcal{D}$  is the unlabelled target data, and  $\tilde{\mathcal{Y}} (\pmb {x})$  is a set of dis
tant supervision labels for an unlabelled input  $\pmb{x} = x_{1}x_{2}\dots x_{n}$ . Thus,  $\tilde{\mathcal{V}} (\pmb {x})$  contains supervision in the form of one or more potentially ambiguous/uncertain labels. In single-source transfer,  $\tilde{\mathcal{V}} (\pmb {x})$  can be as simple as a singleton containing the predicted label for  $\pmb{x}$  by the source model, in which case this is related to self-training (McClosky et al., 2006). In our case, however, this supervision is assumed to arise from an ensemble of models, each is based on transfer from a different source language (Section 2.1). Optionally, the parameters  $\pmb{\theta}$  can be initialised to the source model parameters (or the parameters of one of the source models in the multi-source case) and regularised to this initialiser during training, in order to both speed up training and encourage the parameters to stay near known good parameter values. Overall, the objective becomes  $\ell^{\prime}(\pmb {\theta}) = \ell (\pmb {\theta}) + \lambda \| \pmb {\theta} - \pmb{\theta}_0\| _2^2$  where  $\pmb{\theta}_{0}$  is the source model parameters and  $\lambda$  is a hyperparameter controlling the regularisation strength.
2.1 Supervision via Ensemble
In multi-source transfer, the set  $\tilde{\mathcal{Y}}(\boldsymbol{x})$  can be obtained by an ensemble method applied to the source models. PPTX (Kurniawan et al., 2021) is one such method designed for arc-factored dependency parsers. We generalise PPTX, making it applicable to any set of source models that predict structured outputs that decompose into substructures (of which a set of arc-factored dependency parsers is a special case). For the rest of this paper, we assume that the source models are graphical models over these structured outputs. Let  $C(\boldsymbol{x}, j)$  denote the set of substructures associated with  $x_j$  whose marginal probabilities form a probability distribution:
$$
\sum_ {c \in C (\pmb {x}, j)} p ^ {(k)} (c \mid \pmb {x}) = 1
$$
for any source model  $k$ . For example, for dependency parsing,  $C(\pmb{x},j)$  is the set of arcs whose dependent is  $x_{j}$  (see Fig. 1 part ②). The chart  $\tilde{\mathcal{Y}} (\pmb {x})$  can then be obtained as follows. Define  $\tilde{A}_k(\pmb {x},j)$  to be the set of substructures associated with  $x_{j}$  having high marginal probability under source model  $k$ . This set is obtained by adding substructures  $c\in C(\pmb {x},j)$  in descending order of their marginal probability until their cumulative probability exceeds a threshold  $\sigma$ :
$$
\sum_ {c \in C (\boldsymbol {x}, j)} p ^ {(k)} (c \mid \boldsymbol {x}) \geq \sigma \tag {2}
$$
where  $0 \leq \sigma \leq 1$ . Therefore,  $\tilde{A}_k(x,j)$  contains the substructures that cover at least  $\sigma$  probability mass of the output space under source model  $k$ . Next, define
$$
\tilde {A} (\boldsymbol {x}) = \bigcup_ {k, j} \tilde {A} _ {k} (\boldsymbol {x}, j) \tag {3}
$$
as the set of high probability substructures for  $\pmb{x}$  given by the source models. The chart  $\tilde{\mathcal{V}}(\pmb{x})$  is then defined as the set of structures whose substructures are all in  $\tilde{A}(\pmb{x})$ . Formally,
$$
\tilde {\mathcal {Y}} (\boldsymbol {x}) = \left\{\boldsymbol {y} \mid \boldsymbol {y} \in \mathcal {Y} (\boldsymbol {x}) \wedge A (\boldsymbol {y}) \subseteq \tilde {A} (\boldsymbol {x}) \right\}
$$
where  $\mathcal{V}(\pmb{x})$  is the output space of  $\pmb{x}$  and  $A(\pmb{y})$  is the set of substructures in  $\pmb{y}$ . To prevent  $\tilde{\mathcal{V}}(\pmb{x})$  from being empty, the 1-best structure  $\hat{\pmb{y}} = \arg \max_{\pmb{y}} p^{(k)}(\pmb{y} \mid \pmb{x})$  from each source model  $k$  is also included in the chart, but they don't count toward the probability threshold.
2.2 Proposed Method
Multilinguality is the key factor contributing to the success of PPTX (Kurniawan et al., 2021). Therefore, optimising the method to leverage this multilinguality provided by the source models is important. One potential limitation of PPTX is the inclusion of substructures having relatively low marginal probability under some source model because of the union in Eq. (3). As an extreme illustration, consider a poor source model  $k$  assigning uniform marginal probability to substructures in  $C(\pmb{x}, j)$ . Most of these substructures will be included in  $\tilde{A}_k(\pmb{x}, j)$  and, subsequently,  $\tilde{A}(\pmb{x})$ . As a result, noisy structures may be included in  $\tilde{\mathcal{Y}}(\pmb{x})$  which makes learning the correct structure difficult.
Instead of computing the set of high-probability substructures from each source model separately, a potentially better alternative is to aggregate the marginal probabilities given by the source models and then compute the chart from the resulting distribution. We propose to use logarithmic opinion pooling (Heskes, 1998) as the aggregation method. To obtain the chart  $\tilde{\mathcal{V}}(x)$ , first we compute the logarithmic opinion pool of the source models' marginal probabilities. That is, for all  $j \in \{1, \ldots, n\}$ , define
$$
\bar {p} _ {j} (c \mid \boldsymbol {x}) \propto \prod_ {k} \left[ p ^ {(k)} (c \mid \boldsymbol {x}) \right] ^ {\alpha_ {k}} \tag {4}
$$
where we normalise over the substructures  $c \in C(\pmb{x}, j)$ , and  $\alpha_{k}$  is a non-negative scalar weighting the contribution of source model  $k$  satisfying
Figure 2: Logarithmic opinion pool with uniform weighting  $(\bar{p})$  for two distributions  $p^{(1)}$  and  $p^{(2)}$ . The opinion pool  $\bar{p}$  assigns lower probabilities to substructures indexed by 0 and 1 than those indexed by 3 and 4 because  $p^{(1)}$  and  $p^{(2)}$  assign very low probability to either 0 or 1. Selected substructures in the context of Eq. (2) with  $\sigma = 0.7$  are in orange.
$\sum_{k}\alpha_{k} = 1$  . Thus,  $\bar{p}_j$  gives the new probability distribution over substructures in  $C(\pmb {x},j)$  . Then, we compute the set  $\tilde{A} (\pmb {x},j)$  using  $\bar{p}_j$  in a similar fashion as before: adding substructures  $c\in C(\pmb {x},j)$  in descending order by their marginal probability given by  $\bar{p}_j$  until their cumulative probability exceeds  $\sigma$  . Lastly, we define  $\tilde{A} (\pmb {x}) = \bigcup_{j}\tilde{A} (\pmb {x},j)$  and keep the definition of  $\tilde{\mathcal{V}} (\pmb {x})$  unchanged: the set of structures induced by  $\tilde{A} (\pmb {x})$  plus the 1-best structures,2 which is used as labels for training with the objective in Eq. (1). Fig. 1 illustrates the process using dependency parsing as an example.
Setting the Weight Factors Finding an optimal value for  $\alpha_{k}$  is possible if there is labelled data (Heskes, 1998). However, we do not have labelled data in the target language in our crosslingual setup. There is some method to find similar weighting scalars for cross-lingual transfer that may work in our setup (Wu et al., 2020), but they require unlabelled source language data and only marginally outperform uniform weighting. Therefore, unless stated otherwise, we set  $\alpha_{k}$  uniformly, reducing Eq. (4) to the normalised geometric mean of the marginal distributions.
Motivation The motivation behind the proposed method is the observation that PPTX obtains the high-probability substructures by applying the threshold in Eq. (2) for each source model separately before they are aggregated into a single set in Eq. (3). This means PPTX considers the uncertainty of the source models in isolation to create the chart. In contrast, our method considers the uncertainty of the ensemble by applying the threshold after aggregating the probabilities in the logarithm
2Concrete (sub)structures in set  $C$ ,  $\tilde{A}$ , and  $\tilde{\mathcal{V}}$  depend on the task. For parsing, they are arcs and trees. For tagging, tag pairs and sequences. See Section 2.3 for more details.
mic opinion pool in Eq. (4). The opinion pool assigns more probability mass to substructures to which all the source models assign a high probability (see Fig. 2), and we hypothesise that such substructures are more likely to be correct.
2.3 Application to Structured Prediction
The above method can be applied to structured prediction problems. Crucial to the application is the definition of  $C(\pmb{x}, j)$ . Below, we present two applications: arc-factored dependency parsing and sequence tagging.
Arc-Factored Dependency Parsing For dependency parsing, we can define  $C(\pmb{x}, j)$  as the set of dependency arcs having  $x_j$  as dependent:
$$
C (\pmb {x}, j) = \{(i, j, l) \mid i \in \{0, \dots , n \} \land i \neq j \land l \in L \}
$$
where  $(i,j,l)$  denotes an arc from head  $x_{i}$  to dependent  $x_{j}$  with dependency label  $l$ ,  $L$  denotes the set of dependency labels, and  $x_0$  is a special token whose dependent is the root of the sentence. Since exactly one arc in  $C(\pmb {x},j)$  exists in any possible dependency tree of  $\pmb{x}$ , the marginal probabilities of arcs in  $C(x,j)$  form a probability distribution. The rest follows accordingly.
Sequence Tagging In sequence tagging, the structured output is a sequence of tags, which decomposes into consecutive tag pairs. Given a sequence of tags  $\mathbf{y} = y_{1}y_{2}\dots y_{n}$  corresponding to the input  $\mathbf{x}$ , its consecutive tag pairs are  $A(\mathbf{y}) = \{(j,y_j,y_{j + 1})\}_{j = 1}^{n - 1}$ . We define  $C(\mathbf{x},j)$  as the set of possible tag pairs for  $x_{j}$  and  $x_{j + 1}$ :
$$
C (\boldsymbol {x}, j) = \left\{\left(j, t, t ^ {\prime}\right) \mid \left(t, t ^ {\prime}\right) \in T \times T \right\}
$$
where  $T$  is the set of tags. Note that any sequence of tags for  $\pmb{x}$  has exactly one tag pair in  $C(\pmb{x}, j)$  and thus, the marginal probabilities of these tag pairs in  $C(\pmb{x}, j)$  form a probability distribution.
Data and Evaluation We evaluate on dependency parsing and part-of-speech (POS) tagging. We use Universal Dependencies v2.2 (Nivre et al., 2018) and test on 18 languages spanning 5 language families (see Appendix A). We divide the languages into distant and nearby groups based
on their distance to English (He et al., 2019). We use the universal POS tags (UPOS) as labels for tagging. We exclude punctuation from parsing evaluation following the standard practice and report average performance across five random seeds unless stated otherwise. We also include a PPTX baseline applied to tagging. Our evaluation metric is accuracy for both tasks, which is equivalent to LAS for parsing.
Model Architecture For parsing, we use the same architecture as was used by Kurniawan et al. (2021), consisting of embedding layers, a Transformer encoder layer, and a biaffine output layer (Dozat and Manning, 2017). At test time, we run the MST algorithm (Chu and Liu, 1965; Edmonds, 1967) to find the highest scoring tree. For tagging, we replace the output layer with a linear CRF layer. At test time, the Viterbi algorithm is used to obtain the tag sequence with the highest score.
Source Selection We adopt a "pragmatic" approach where we include 5 high-resource languages as sources: English, Arabic, Spanish, French, and German (Kurniawan et al., 2021), which have been categorised as "quintessential rich resource languages" due to the availability of massive language datasets (Joshi et al., 2020). When a source language is also the target language, we exclude the language from the sources. For example, if Arabic is the target language, then we use only the other 4 languages as sources, thus the target language is always unseen. See Appendix B for more details.
Baselines Our main baseline for both tasks is a majority voting ensemble (MV). For parsing, we score each possible arc by the number of source parsers that have the arc in the predicted tree and then run the MST algorithm. For tagging, we use the most commonly predicted tag for each input token. This baseline is not only more appropriate for multi-source transfer but also stronger than the direct transfer baseline used by Kurniawan et al. (2021) which uses only a single source language (English), with accuracy gains of up to 15 points on both tasks. We also include knowledge distillation (KD) which has been used for parsing as a baseline (Hu et al., 2021). For tagging, we include BEA (Rahimi et al., 2019) which explicitly
This setup is called PPTX-PRAG by Kurniawan et al. (2021), which is reported in their Figure 3.
3This formulation is widely used in graph-based dependency parsing, which dates back to the work of McDonald et al. (2005).
(a) Dependency parsing
(b) POS tagging
Figure 3: Performance difference of PPTX and our method against majority voting (MV) on parsing and tagging. Numbers on the x-axis are the MV performance corresponding to the zero value on the y-axis. BEA is run only once as it always gives the same result. Table 11 reports the full results. \*: hyperparameters are tuned on this language.
models the label confusion of source taggers and has been used successfully for multi-source crosslingual NER. More details on these two baselines are reported in Appendix C.
Training We use the same setup as Kurniawan et al. (2021) for parsing. We include the gold universal POS tags as input to the parsers. We discard sentences longer than 30 tokens to avoid memory issues and train for 5 epochs using Adam (Kingma and Ba, 2015). Note that we discard long sentences only at training time. In other words, we evaluate on all lengths at test time. We tune the learning rate and  $\lambda$  on the development set of Arabic, select the values that give the highest accuracy, and use them for training on all languages. For tagging, we set the length cut-off to 60 tokens (again, only at training time) and train for 10 epochs. Again, we tune the hyperparameters on Arabic and use the best values for training on all languages. For both tasks, we obtain cross-lingual word embeddings using an offline transformation method (Smith et al., 2017) applied to fastText pre-trained word vectors (Bojanowski et al., 2017). We set the threshold  $\sigma = 0.95$  (Kurniawan et al., 2021). We initialise the parameters of the target language model with the parameters of the English source model and regularise the former towards the latter during training. In other words, we set the parameters of the English source model as  $\theta_0$  as described in Section 2.1. See Appendix D for further details.
Lang.	Parsing	Tagging
nP(millions)	nQ/nP(%)	nP	nQ/nP(%)
fa	1.6 × 106	0.0011	6.5 × 105	3
ko	2.3 × 104	0.021	8.2 × 103	11
hr	2.0 × 105	0.0019	4.3 × 105	37
it	4.5	0.069	4.7 × 104	32
es	3.7 × 103	0.0014	2.4 × 106	110
sv	5.1	0.12	7.6 × 103	18
Table 1: Median chart size of PPTX (column  $n_P$ ), and median chart size of our method relative to PPTX (column  $\frac{n_Q}{n_P}$ ), where chart size is defined as the number of structures in  $\tilde{\mathcal{V}}(\boldsymbol{x})$ .
Fig. 3a shows the accuracy difference of KD, PPTX, and our method against MV on parsing. We see that PPTX does not consistently outperform MV, substantially underperforming on 6 languages. On the other hand, our method outperforms not only PPTX but also both KD and MV on most languages. Fig. 3b shows the corresponding results on POS tagging which is particularly convincing. We see that PPTX often underperforms, with up to  $10\%$  drop in accuracy compared to MV. In contrast, our method consistently outperforms MV with up to  $10\%$  accuracy improvement. These results suggest that PPTX may not improve over a
Persian, Arabic, Indonesian, Turkish, Italian, and Portuguese.
Method	Distant	Nearby
PPTX	-1.2 ± 5.0	0.5 ± 2.4
Ours	2.1 ± 2.7	2.8 ± 2.0
Table 2: LAS difference against majority voting using predicted POS tags, averaged over all nearby and distant languages (± std). Average LAS for the majority voting on distant and nearby languages are  $38.4 \pm 18.3$  and  $68.1 \pm 7.3$  respectively.
simple majority voting ensemble, and our method is the superior alternative. In addition, our method shows higher improvement against MV on nearby than distant languages, which is unsurprising because our pragmatic selection of source languages is dominated by languages in the nearby group.
From the figure, we also see that on Portuguese and Italian, our method slightly underperforms compared to MV on parsing, but outperforms MV considerably on tagging. We hypothesise that this disparity is caused by the variability of the source models quality. On tagging, the direct transfer performance of 3 out of 5 source taggers is relatively poor on Portuguese and Italian, making it more likely for MV to predict wrongly as the good taggers are outvoted. In contrast, on parsing, Arabic is the only source parser that has very poor transfer. The other source parsers achieve comparably good direct transfer performance so MV already performs well.
Parsing results using predicted POS tags Since low-resource languages often don't have gold POS tags, we also evaluate our method for parsing using predicted POS tags. We use Stanza (Qi et al., 2020) to predict the POS tags of all target languages, and replace the gold UPOS with the predicted tags as the input. Table 2 shows that our method still outperforms PPTX in this setup, although there is a large variance across languages.
4.1 Chart Size Analysis
To understand the differences between PPTX and our method better, we compare the size of the chart  $\tilde{\mathcal{V}}(\boldsymbol{x})$  produced by PPTX and our method, in terms of the number of structures in it. We take the median of this size over all unlabelled sentences in the training set of each target language and compare the results. Table 1 reports the median chart size of PPTX, and the median chart size of our method relative to PPTX for both parsing and tagging on