SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\begin{array}{l} \ell (\boldsymbol {\theta}) = \operatorname {M S E} \left(p _ {\text {h e a d}} (\cdot \mid \boldsymbol {x}; \boldsymbol {\theta}), \bar {p} _ {\text {h e a d}} (\cdot \mid \boldsymbol {x})\right) \\ + \tau \operatorname {M S E} \left(p _ {\text {l a b e l}} (\cdot \mid \boldsymbol {x}; \boldsymbol {\theta}), \bar {p} _ {\text {l a b e l}} (\cdot \mid \boldsymbol {x})\right) \\ + \lambda \| \boldsymbol {\theta} - \boldsymbol {\theta} _ {0} \| _ {2} ^ {2} \\ \end{array}
$$
where MSE denotes the mean squared error function. We include  $\tau$  as a tunable hyperparameter. Table 9 reports the full list of hyperparameter values.
Distant	Nearby
fa	ar	id	ko	tr	hi	hr	he	bg	it	pt	fr	es	no	da	sv	nl	de
Parser	18	19	19	49	20	19	23	18	22	20	24	23	28	30	19	19	24	26
Tagger	16	18	18	48	18	18	23	17	21	20	25	24	31	29	17	17	23	26
Table 7: Number of parameters of the parsers and taggers with our method, rounded to the nearest million.
BEA We use the implementation of BEA (Rahimi et al., 2019) provided by the authors. We run BEA on the unlabelled test data of each language (transductive setup). This potentially gives the BEA baseline more advantage, so it is not directly comparable to PPTX and our method. We find that our hyperparameter tuning protocol (i.e. tuning the parameters of the Dirichlet priors on Arabic and using the best values for all languages) underperforms compared to using uninformative priors, so we report the BEA results without any tuning.
D Additional Experiment Details
en	ar	es	fr	de
Parser	14	14	27	20	23
Tagger	12	12	25	18	21
We implement our method using Python v3.7, PyTorch v1.4 (Paszke et al., 2019), and PyTorch-Struct (Rush, 2020). We run our experiments with Sacred v0.8.2 (Greff et al., 2017), which also sets the random seeds. Experiments are run on NVIDIA GeForce GTX TITAN X with CUDA 10.1 and GPU memory of 11 MiB. CPU model is Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz with Ubuntu 16.04 as the operating system. Table 8 and Table 7 show the number of parameters of the source parsers and taggers, and the target parsers and taggers using our method respectively. A single run takes not much longer than 1 GPU hour for both parsing and tagging.
We tune learning rate  $\eta$  and  $\lambda$  (and also  $\tau$  for KD) using random search. Table 9 shows the distributions of each hyperparameter we use, and the best values we find. We sample 20 values from the
Table 8: Number of parameters of the source parsers and taggers, rounded to the nearest million.
Task	Method	Hyperparameter Dist.	Best Value
Parsing	KD	log η ~ U(-6, -3)	η = 3.3 × 10-5
log τ ~ U(-2, 2)	τ = 0.66
log λ ~ U(-4, 1)	λ = 10-3
PPTX	log η ~ U(-6, -3)	η = 8.5 × 10-5
log λ ~ U(-4, 1)	λ = 2.8 × 10-5
Ours	log η ~ U(-6.5, -3.5)	η = 9.4 × 10-5
log λ ~ U(-4, 1)	λ = 1.6 × 10-4
Tagging	PPTX	log η ~ U(-6, -4)	η = 5.9 × 10-5
log λ ~ U(-4, 1)	λ = 0.1
Ours	log η ~ U(-6.5, -3.5)	η = 2.6 × 10-4
log λ ~ U(-4, 1)	λ = 4.7 × 10-3
Table 9: Distributions of hyperparameters we use for tuning on Arabic with random search and the best values found. All logarithms are of base 10.
Hyperparameter	Value
Word embedding size	300
Word dropout	0.2
dkey, dvalue	64
df	512
nhad	8
nlayer	6
Batch size	80
Parsing-only
POS tag embedding size	50
Output embedding dropout	0.2
darc	512
ddeptype	128
Table 10: List of hyperparameter values used in our parsers and taggers.  $d_{\mathrm{key}}$ ,  $d_{\mathrm{value}}$ : size of key and value vector in the Transformer encoder.  $d_{\mathrm{ff}}$ : size of feedforward network hidden layer in the Transformer encoder.  $n_{\mathrm{head}}$ : number of heads in the Transformer encoder.  $n_{\mathrm{layer}}$ : number of layers in the Transformer encoder.  $d_{\mathrm{arc}}$ ,  $d_{\mathrm{deptype}}$ : size of feedforward network output layer corresponding to arcs and dependency types in the bi-affine output layer of parsers.
distribution and pick the values that yield the best accuracy on the Arabic development set. We follow Kurniawan et al. (2021) for other hyperparameters, whose values are reported in Table 10.
<sup>9</sup>https://github.com/afshinrahimi/mmner
F Learning the Opinion Pool Weight Factors
We learn the factors  $\alpha_{k}$  weighting the contribution of source model  $k$  in the logarithmic opinion pool by minimising Eq. (5) with respect to  $\alpha_{k}$ . The minimisation is done on 50 randomly sampled sentences from the target language's training set using gradient descent. We set the initial learning rate to 0.1 and reduce it at every epoch by a factor of 0.9. We initialise the weight factors uniformly at the start and run the training until convergence. After the weight factors are learned, we use and fix them for all subsequent experiments. We proceed with hyperparameter tuning on Arabic using the same procedure as the version with uniform weights. For both tasks, we tune  $\eta$  and  $\lambda$  with random search (20 runs), drawing from  $\log_{10} \eta \sim U(-6, -3)$  and  $\log_{10} \lambda \sim U(-4, 1)$  respectively. For parsing, the best values are  $\eta = 9.1 \times 10^{-5}$  and  $\lambda = 5.1 \times 10^{-4}$ . For tagging, they are  $\eta = 4.7 \times 10^{-4}$  and  $\lambda = 0.062$ . These values are then used for the other languages. Lastly, we report the average accuracy over the languages in Table 4.
G Full Experiment Results
We report in Table 11 the full results of MV, PPTX, and our method (with both uniform and learned weight factors  $\alpha_{k}$ ) on both dependency parsing and POS tagging, averaged over 5 runs.
Lang.	Parsing	Tagging
MV	KD	PPTX	Ours	Ours, learned αk	MV	BEA	PPTX	Ours	Ours, learned αk
Distant
fa	43.7	42.3 ± 0.2	42.5 ± 1.1	49.4 ± 0.5	48.9 ± 0.3	69.2	67.6	67.5 ± 0.2	71.2 ± 0.6	72.3 ± 0.5
ar*	37.6	37.5 ± 0.2	36.4 ± 0.6	38.9 ± 0.5	38.6 ± 0.5	58.5	57.5	59.0 ± 0.4	62.5 ± 1.1	63.0 ± 1.7
id	56.8	57.5 ± 0.4	54.8 ± 0.8	59.0 ± 0.3	59.1 ± 0.1	77.5	80.0	79.6 ± 0.4	81.0 ± 0.2	80.6 ± 0.8
ko	13.7	15.2 ± 0.3	13.6 ± 0.4	12.8 ± 0.3	13.6 ± 0.2	44.1	41.1	44.0 ± 0.3	44.2 ± 1.1	43.4 ± 1.7
tr	20.8	21.0 ± 0.2	19.9 ± 0.2	20.2 ± 0.4	21.2 ± 0.2	62.8	63.3	62.8 ± 0.3	64.2 ± 0.3	64.3 ± 0.2
hi	21.9	23.0 ± 0.1	23.9 ± 0.5	22.7 ± 0.2	27.2 ± 0.2	59.9	55.1	65.6 ± 0.2	63.2 ± 0.6	65.5 ± 0.9
hr	57.1	59.6 ± 0.1	60.7 ± 0.3	62.9 ± 0.4	62.8 ± 0.3	67.2	64.9	58.6 ± 0.1	69.4 ± 0.3	69.7 ± 0.3
he	56.1	56.6 ± 0.1	58.1 ± 0.5	60.6 ± 0.2	60.0 ± 0.2	56.3	56.0	57.6 ± 0.1	59.9 ± 0.2	58.8 ± 0.5
Nearby
bg	69.3	70.7 ± 0.2	71.9 ± 0.2	72.7 ± 0.4	72.5 ± 0.4	75.0	76.7	75.9 ± 0.2	76.7 ± 0.2	76.1 ± 0.2
it	81.5	79.0 ± 0.2	80.1 ± 0.2	81.7 ± 0.2	81.5 ± 0.1	74.7	78.9	62.8 ± 0.7	84.8 ± 0.3	85.5 ± 0.7
pt	78.6	75.5 ± 0.1	76.2 ± 0.4	78.1 ± 0.3	78.4 ± 0.3	72.0	68.0	63.5 ± 0.9	81.7 ± 0.4	83.2 ± 1.0
fr	80.0	79.8 ± 0.1	81.3 ± 0.2	82.7 ± 0.2	82.8 ± 0.1	65.7	58.2	54.9 ± 0.5	71.7 ± 0.7	76.3 ± 0.7
es	71.8	70.9 ± 0.1	72.0 ± 0.6	73.5 ± 0.3	73.7 ± 0.2	67.8	68.7	58.1 ± 0.2	73.9 ± 0.7	75.7 ± 2.1
no	68.4	71.8 ± 0.2	74.1 ± 0.2	74.2 ± 0.1	74.4 ± 0.2	62.2	61.0	61.4 ± 0.2	64.7 ± 0.5	64.6 ± 1.1
da	67.5	68.9 ± 0.1	70.4 ± 0.4	71.0 ± 0.1	70.9 ± 0.3	72.9	73.3	72.0 ± 0.1	76.0 ± 0.3	76.2 ± 0.7
sv	66.7	69.7 ± 0.1	71.8 ± 0.2	72.1 ± 0.1	72.4 ± 0.5	68.4	69.5	68.5 ± 0.1	69.0 ± 0.4	70.7 ± 0.6
nl	64.8	64.4 ± 0.2	66.9 ± 0.2	67.4 ± 0.4	68.8 ± 0.4	72.9	75.1	70.3 ± 0.3	74.3 ± 0.4	75.3 ± 0.7
de	57.2	62.4 ± 0.2	64.0 ± 0.9	64.0 ± 0.5	63.9 ± 0.5	52.8	62.5	59.3 ± 0.3	58.9 ± 0.5	58.0 ± 0.4
(a) Development set
Lang.	Parsing	Tagging
MV	KD	PPTX	Ours	Ours, learned αk	MV	BEA	PPTX	Ours	Ours, learned αk
Distant
fa	43.7	42.3 ± 0.2	42.5 ± 1.0	49.4 ± 0.4	48.8 ± 0.3	69.4	66.0	67.4 ± 0.3	71.1 ± 0.7	72.5 ± 0.7
ar*	37.3	36.6 ± 0.2	35.5 ± 0.5	37.3 ± 0.5	37.2 ± 0.6	57.8	58.2	59.0 ± 0.5	63.0 ± 1.4	63.3 ± 1.6
id	59.0	59.5 ± 0.3	57.4 ± 0.6	61.6 ± 0.2	61.4 ± 0.2	77.9	80.3	79.9 ± 0.4	81.0 ± 0.2	80.8 ± 0.8
ko	14.7	16.6 ± 0.3	14.8 ± 0.5	13.8 ± 0.4	14.7 ± 0.1	45.0	42.6	45.7 ± 0.2	45.9 ± 1.1	44.9 ± 1.5
tr	20.1	20.6 ± 0.1	19.3 ± 0.2	19.7 ± 0.3	20.8 ± 0.2	62.8	63.2	63.0 ± 0.2	64.7 ± 0.5	64.9 ± 0.1
hi	21.1	22.2 ± 0.1	23.0 ± 0.5	21.7 ± 0.2	26.4 ± 0.3	59.7	59.9	65.1 ± 0.2	62.8 ± 0.5	65.1 ± 1.0
hr	57.4	61.1 ± 0.1	62.2 ± 0.2	64.3 ± 0.4	64.2 ± 0.3	66.7	66.6	58.4 ± 0.1	68.5 ± 0.2	69.0 ± 0.4
he	56.2	56.8 ± 0.3	57.5 ± 0.7	60.1 ± 0.2	59.7 ± 0.3	55.5	56.9	57.2 ± 0.1	59.1 ± 0.4	58.0 ± 0.4
Nearby
bg	69.3	70.9 ± 0.1	72.4 ± 0.2	73.1 ± 0.3	72.9 ± 0.3	75.6	76.4	76.2 ± 0.1	77.1 ± 0.3	76.6 ± 0.2
it	81.7	79.8 ± 0.1	80.2 ± 0.1	81.4 ± 0.2	81.4 ± 0.3	74.5	81.7	62.1 ± 0.8	85.0 ± 0.4	86.0 ± 0.8
pt	76.6	73.7 ± 0.1	74.5 ± 0.4	76.3 ± 0.3	76.5 ± 0.3	72.5	69.6	63.4 ± 0.8	81.8 ± 0.5	83.1 ± 0.9
fr	76.5	76.2 ± 0.1	78.2 ± 0.3	79.3 ± 0.1	79.1 ± 0.1	65.4	60.2	56.0 ± 0.4	72.2 ± 0.6	75.7 ± 0.5
es	71.3	70.3 ± 0.2	71.5 ± 0.6	73.1 ± 0.4	73.2 ± 0.3	67.1	72.7	57.8 ± 0.2	73.5 ± 0.6	75.1 ± 2.0
no	69.1	72.0 ± 0.1	74.1 ± 0.2	74.2 ± 0.2	74.5 ± 0.2	63.2	62.4	62.3 ± 0.3	65.9 ± 0.6	65.7 ± 1.1
da	67.3	69.0 ± 0.1	70.7 ± 0.3	71.3 ± 0.1	71.2 ± 0.4	73.9	75.0	72.8 ± 0.1	77.1 ± 0.4	77.3 ± 0.6
sv	70.1	72.7 ± 0.1	74.5 ± 0.3	74.7 ± 0.2	75.0 ± 0.3	69.8	69.3	69.6 ± 0.1	70.0 ± 0.3	72.0 ± 0.5
nl	65.8	65.9 ± 0.2	67.8 ± 0.4	68.5 ± 0.3	69.6 ± 0.4	70.8	68.9	68.7 ± 0.3	73.0 ± 0.3	74.5 ± 1.0
de	55.3	60.8 ± 0.3	61.6 ± 0.8	61.9 ± 0.6	61.7 ± 0.5	50.2	59.5	56.4 ± 0.3	55.8 ± 0.6	54.9 ± 0.4
(b) Test set
Table 11: Full performance results. Except for MV and BEA, numbers are averages (± std) over 5 runs with different random seeds. For parsing, the numbers correspond to labelled attachment score (LAS) whereas for tagging, they correspond to accuracy. Both metrics are better if higher. Hyperparameters are tuned on Arabic, hence the asterisk. In columns "Ours, learned  $\alpha_{k}$ ",  $\alpha_{k}$  is learned in a supervised manner on tiny labelled sentences (Section 4.4).