SlowGuess/NAACL_2022_OCR huggingface dataset:
We define a few quantities to express the transformation mathematically. Let  $s_i = (w_{i1},\ldots ,w_{ik})$  be a sentence comprised of  $k$  tokens and  $y_{i} = (y_{i1},\dots ,y_{ik})$  be labels corresponding to the tokens in the sentence. We define a new transformation  $(\mathcal{T}_{aug})$  which operates on the label augmented sentence,  $s_i^{aug} = ((w_{i1},y_{i1}),\ldots ,(w_{ik},y_{ik}))$ . Let  $s_i^{aug}[j]$  correspond to the  $j^{th}$  element in the sequence, and  $s_i^{aug}[j][word]$  and  $s_i^{aug}[j][label]$  correspond to the word and label of the  $j^{th}$  element. Let  $\mathcal{T}_{aug}(s_i^{aug})[j][orig]$  denote the index of the  $j^{th}$  element in the transformed sequence with respect to the original sequence  $s_i^{aug}$ . Then, the new transformation  $\mathcal{T}_{aug}$  is such that,
$$
\mathcal {T} _ {a u g} \left(s _ {i} ^ {a u g}\right) [ j ] [ o r i g ] = \mathcal {T} \left(s _ {i}\right) [ j ] [ o r i g ]
$$
Let ori g  $\mathbf{j} = \mathcal{T}_{\mathrm{aug}}(s_i^{\mathrm{aug}})[j][\mathrm{orig}]$
$$
\mathcal {T} _ {a u g} (s _ {i} ^ {a u g}) [ j ] [ \text {l a b e l} ] = s _ {i} ^ {a u g} [ \text {o r i g \_ j} ] [ \text {l a b e l} ]
$$
We transform the dataset using  $\mathcal{T}_{\text{aug}}$ :
$$
\mathcal {D} _ {\text {o r i g}} = \left\{s _ {i} ^ {a u g} \mid i = 1: N \right\}
$$
$$
\mathcal {D} _ {\text {d e r i v}} = \left\{\mathcal {T} _ {a u g} \left(s _ {i} ^ {a u g}\right) \mid i = 1: N \right\}
$$
B Zero-shot transfer results for different transformations
Table 5 in the appendix is the extended version of Table 3 in the main paper with a breakdown for all languages. It reports  $\Delta_{\mathrm{(BZ - BS)}}$ ,  $\Delta_{\mathrm{(MZ - BS)}}$ , and  $BZ$  for different languages and transformations considered.
C Composed Transformations
Table 6 in the appendix presents the breakdown of results in Figure 2 of the main paper. It reports  $\Delta_{\mathrm{(BZ - BS)}}$  scores for composed transformations and their constituents.
D Comparing different sources for original and derived language corpora
Table 8 in the appendix contains the breakdown of results in Figure 3 of the main paper. It reports  $\Delta_{\mathrm{(BZ - BS)}}$  for different languages on different tasks for the settings mentioned in Section 4.3.
E Alignment Correlation
We present alignment results (Section 4.4) for all XNLI, NER, and POS in Figure 5. We observe strong correlations between alignment and zero-shot transfer, with  $\rho_{s} = 0.94$ ,  $p < .005$  on XNLI,  $\rho_{s} = 0.93$ ,  $p < .005$  on NER, and  $\rho_{s} = 0.89$ ,  $p < .01$  on POS. We present the raw scores in Table 7.
F Hyperparameters for XTREM
- XNLI: Learning rate  $-2\mathrm{e} - 5$ , maximum sequence length - 128, epochs - 5, batch size - 32.
- NER: Learning rate  $-2 \mathrm{e} - 5$ , maximum sequence length  $-128$ , epochs  $-10$ , batch size  $-32$ .
- POS: Learning rate  $-2 \mathrm{e} - 5$ , maximum sequence length - 128, epochs - 10, batch size - 32.
- Tatoeba: Maximum sequence length - 128, pooling strategy - representations from the middle layer  $\left(\frac{n}{2}\right)$  of the model.
XQuAD: Learning rate  $-3 \mathrm{e} - 5$ , maximum sequence length - 384, epochs - 2, document stride - 128, warmup steps - 500, batch size - 16, weight decay - 0.0001.
Task	Language	Inversion	Permutation	Syntax	Transliteration
\(BZ\)	\(\Delta_{(BZ-BS)}\)	\(\Delta_{(MZ-BS)}\)	\(BZ\)	\(\Delta_{(BZ-BS)}\)	\(\Delta_{(MZ-BS)}\)	\(BZ\)	\(\Delta_{(BZ-BS)}\)	\(\Delta_{(MZ-BS)}\)	\(BZ\)	\(\Delta_{(BZ-BS)}\)	\(\Delta_{(MZ-BS)}\)
XNLI	English	73.2	-3.4	-14.9	68.6	-5	-7.7	74.1	-1.8	-1.5	74.1	-1.7	-42.5
French	62.5	-9.5	-8.8	68.4	-1	-7.6	69.6	-2.2	-1.4	71.6	-1.6	-39.9
Hindi	43.9	-15.7	-15.8	51.2	-6.2	-13.1	61.6	-0.3	-1.6	63.4	-0.1	-29.4
Arabic	54	-12.3	-12.5	62.1	-2.3	-6	65.9	0.7	0.3	68	-0.4	-35.1
Avg.	58.4	-10.2	-13	62.6	-3.6	-8.6	67.8	-0.9	-1.1	69.3	-1.0	-36.7
NER	English	39.8	-44.5	-35.9	40.2	-28.5	-33.2	61.1	-7.8	-10.3	78	-2.1	-70.2
French	54.5	-34.4	-51.3	44.4	-36.0	-39.8	59.6	-21.9	-25.9	84.3	-3.1	-87.4
Hindi	19.4	-63.9	-63.2	38.5	-21.9	-37.4	64.8	-8.4	-7.3	84.4	-0.5	-82.9
Arabic	37.8	-53.6	-36.3	66.2	-18.8	-31.1	66.1	-20.1	-23	88	-1.9	-89.9
Avg.	37.9	-49.1	-46.7	47.3	-26.3	-35.4	62.9	-14.6	-16.6	83.7	-1.9	-82.6
POS	English	94.4	-0.7	-24.3	78.3	-11.9	-17.6	92.9	-0.9	-2.2	94.6	-0.5	-95.1
French	74.3	-22.7	-22.9	82	-12.2	-20.9	93.5	-3.2	-5.2	97.2	-0.2	-97.4
Hindi	19	-74.5	-74.5	51	-14	-41.8	91.6	-3.3	-11.3	96.5	-0.1	-96.6
Arabic	69.2	-23	-23	83.1	-6.5	-20.6	79.4	-10	-11.5	93.2	-0.8	-90.9
Avg.	64.2	-30.2	-36.2	73.6	-11.2	-25.2	89.4	-4.4	-7.6	95.4	-0.4	-95.0
XQuAD	English	30.4	-43.2	-35.5	-	-	-	-	-	-	72.4	-4	-73
French	25.2	-29.5	-29.6	-	-	-	-	-	-	60.9	-1	-55.5
Hindi	14.5	-27.3	-27.3	-	-	-	-	-	-	57.3	10.6	-43.5
Arabic	21	-31.2	-31.4	-	-	-	-	-	-	54	-0.5	-51.7
Avg.	22.8	-32.8	-31.0	61.2	1.3	-55.9
Table 5: This table is an extended version of Table 3 in the main paper. Smaller (more negative)  $\Delta_{\mathrm{(BZ - BS)}}$  implies worse bilingual zero-shot transfer, whereas  $\Delta_{\mathrm{(BZ - BS)}} \approx 0$  implies strong transfer.  $\Delta_{\mathrm{(BZ - BS)}} \gg \Delta_{\mathrm{(MZ - BS)}}$  implies that bilingual pre-training is extremely useful. Scores are highlighted based on their value (lower scores have a higher intensity of red). (1) Discussing  $\Delta_{\mathrm{(BZ - BS)}}$ :  $\mathcal{T}_{\mathrm{trans}}$  exhibits strong transfer on all tasks and languages (high  $\Delta_{\mathrm{(BZ - BS)}}$  scores), and bilingual pre-training is extremely useful ( $\Delta_{\mathrm{(BZ - BS)}} \gg \Delta_{\mathrm{(MZ - BS)}}$ ), implying that zero-shot transfer is possible between languages with different scripts but the same word order.  $\mathcal{T}_{\mathrm{inv}}$  and  $\mathcal{T}_{\mathrm{perm}}$  suffer on all tasks (small  $\Delta_{\mathrm{(BZ - BS)}}$  scores) whereas  $\mathcal{T}_{\mathrm{syn}}$  suffers significantly lesser, which provides evidence that local changes to the word order made by Syntax ( $\mathcal{T}_{\mathrm{syn}}$ ) hurts zero-shot transfer significantly lesser than global changes made by Inversion ( $\mathcal{T}_{\mathrm{inv}}$ ) and Permutation ( $\mathcal{T}_{\mathrm{perm}}$ ). (1) Discussing  $\Delta_{\mathrm{(MZ - BS)}}$ :  $\Delta_{\mathrm{(BZ - BS)}}$  is much larger than  $\Delta_{\mathrm{(MZ - BS)}}$  for  $\mathcal{T}_{\mathrm{trans}}$ , implying that bilingual pre-training (hereon, pre-training) is extremely useful.  $\Delta_{\mathrm{(BZ - BS)}}$  and  $\Delta_{\mathrm{(MZ - BS)}}$  are similar for  $\mathcal{T}_{\mathrm{inv}}$  and  $\mathcal{T}_{\mathrm{syn}}$ , implying that pre-training is not beneficial for these transformations.  $\Delta_{\mathrm{(BZ - BS)}}$  is slightly larger than  $\Delta_{\mathrm{(MZ - BS)}}$  for  $\mathcal{T}_{\mathrm{perm}}$ , which means that pre-training is moderately useful.
T	XNLI	NER	POS
BZ	Δ(BZ-BS)	BZ	Δ(BZ-BS)	BZ	Δ(BZ-BS)
Trans	74.1	-2.1	78	-2.3	94.6	-0.5
Tinv	73.2	-3.4	39.8	-44.5	94.4	-0.7
Trans○Tinv	55.7	-19.2	32.5	-51.5	52.2	-42.7
Tperm	68.6	-5	40.2	-28.5	78.3	-11.9
Trans○Tperm	44	-27.7	17.1	-46.3	29.5	-59
Tsyn	74.1	-1.8	61.1	-7.8	92.9	-0.9
Trans○Tsyn	69.8	-5.7	53.5	-14.2	91.5	-2
Table 6: Breakdown of results in Figure 2 of the main paper.  $BZ$  is the zero-shot performance.  $\Delta_{(\mathrm{BZ - BS})}$ ,  $\Delta_{(\mathrm{MZ - BS})}$ , and  $BZ$  are described in Section 3.3 and Table 2. Composing transformations always hurts  $\Delta_{(\mathrm{BZ - BS})}$  when compared to individual transformations.
Transliteration Variant	Δ(BZ-BS) (↑)	Alignment (↑)
XNLI	NER	POS
Parallel	-2.1	-2.3	-0.5	90.0
Trans o Syntax	-5.7	-14.2	-2	57.3
Non-parallel (Same)	-3.8	-4.1	-0.7	43.0
Non-parallel (Diff)	-5.7	-14.3	-1.5	11.8
Trans o Inv	-19.2	-51.5	-42.7	0.16
Trans o Perm	-27.7	-46.3	-59	0.01
Table 7:  $\Delta_{\mathrm{(BZ - BS)}}$  and alignment scores for different Transliteration variants. The table contains raw scores for results in Section 4.4 of the main paper. Rows are sorted in descending order based on alignment. We observe strong correlations between alignment and zero-shot transfer, with  $\rho_{s} = 0.94, p < .005$  on XNLI,  $\rho_{s} = 0.93, p < .005$  on NER, and  $\rho_{s} = 0.89, p < .01$  on POS.
Task	Language	XNLI	NER	POS	XQuAD
Δ(BZ-BS)	Δ(BZ-BS)	Δ(BZ-BS)	Δ(BZ-BS)
Parallel	English	-1.7	-2.1	-0.5	-4
French	-1.6	-3.1	-0.2	-1
Hindi	-0.1	-0.5	-0.1	10.6
Arabic	-0.4	-1.9	-0.8	-0.5
Non-parallel (Same)	Avg.	-1.0	-1.9	-0.4	1.3
English	-3.8	-4.1	-0.7	-6.9
French	-1	-6.3	-0.5	-0.9
Hindi	-0.4	-3.1	-0.2	4.5
Arabic	-2	-6.1	-1.5	0.7
Non-parallel (Diff)	Avg.	-1.8	-4.9	-0.7	-0.6
English	-5.7	-14.3	-1.5	-9.3
French	-10.9	-30.3	-10.5	-5.2
Hindi	-0.5	-8.6	-1	5
Arabic	-6.3	-34.7	-3.7	-1.9
Avg.	-5.9	-22.0	-4.2	-2.9
Table 8:  $|\Delta_{(\mathrm{BZ} - \mathrm{BS})}|$  for  $\mathcal{T}_{\mathrm{trans}}$  under different conditions on the source of original and derived language pretraining corpora (§ 4.3). Larger values imply worse zero-shot transfer. For all languages: (1) Non-parallel (diff), which uses corpora from different domains is worse than (2) Non-parallel (same), which uses different sets of sentences sampled from the same domain, which is in turn worse than (3) Parallel, which uses the same sentences.
Figure 5: Alignment v.s.  $\Delta_{\mathrm{(BZ - BS)}}$  plots for XNLI, NER, and POS. We observe strong correlations between alignment and zero-shot transfer, with  $\rho_{s} = 0.94, p < .005$  on XNLI,  $\rho_{s} = 0.93, p < .005$  on NER, and  $\rho_{s} = 0.89, p < .01$  on POS.