SlowGuess/NAACL_2022_OCR huggingface dataset:
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
George Zerveas, Navid Rekabsaz, Daniel Cohen, and Carsten Eickhoff. 2022. Mitigating bias in search results through set-based document reranking and neutrality regularization. In Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2022. ACM.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568-1575, Austin, Texas. Association for Computational Linguistics.
A Grid search over  $k$  and  $\tau$
To choose number of neighbors  $k$  and temperature  $\tau$  for WECHSEL we conduct a grid search over linear probes of models with different initialization shown in Table 7. For RoBERTa, we compute scores on NLI (using XNLI) and POS tagging (using the French, German and Chinese GSD corpora in Universal Dependencies) using linear probes of the last hidden state. We probe on NLI by taking a concatenation of the mean of all token representations in the premise with the mean of all token representations in the hypothesis. We probe on POS tagging by taking the mean of all token representations belonging to each word. For GPT2, we compute Language Modelling Perplexity on the held-out set also used to evaluate performance of the trained models.
Hyperparameters used to fine-tune RoBERTa on downstream tasks are shown in Table 5. Hyperparameters used to train models in our main experiments are shown in Table 6.
Parameter	NLI	NER
peak learning rate	2e-5	2e-5
batch size	128	32
sequence length	128	128
Adam ε	1e-8	1e-8
Adam β1	0.9	0.9
Adam β2	0.999	0.999
train epochs	2	10
warmup	10% of steps	10% of steps
warmup schedule	linear	linear
LR decay	linear to zero	linear to zero
Table 5: Hyperparameters used to fine-tune RoBERTa models on NLI (XNLI) and NER (WikiANN).
Parameter	RoBERTa	GPT2
peak learning rate	1e-4	5e-4
batch size	512	512
sequence length	512	512
weight decay	0.01	0.01
Adam ε	1e-6	1e-6
Adam β1	0.9	0.9
Adam β2	0.98	0.98
train steps	250k	250k
C Qualitative subword correspondence
We show a small random sample of tokens in the target language and their closest English token (according to WECHSEL) in Table 8.
D Using Word Embeddings without subword information
As an alternative to n-gram decomposition, we introduce a method for mapping word embeddings to subword embeddings without using any subword information (shown in Figure 5). For this method, we require word frequency information in addition to the word embeddings. We apply the tokenizer  $T$  to every word  $v$  in  $\mathbb{V}$  resulting in a set of subwords for each word. We define  $\mathbb{V}^{(x)}$  as the set of words containing the subword  $x$  when tokenized. The embedding  $u_{x}$  of the subword  $x$  is then defined as the average of the embeddings of words in  $\mathbb{V}^{(x)}$ , weighted by the word frequencies.
$$
\boldsymbol {u} _ {x} = \frac {\sum_ {v \in \mathbb {V} ^ {(x)}} \boldsymbol {w} _ {v} \cdot f _ {v}}{\sum_ {v \in \mathbb {V} ^ {(x)}} f _ {v}}
$$
where  $\pmb{w}_v$  is the embedding and  $f_v$  is the frequency of word  $v$ .
Table 6: Hyperparameters of the models transferred from RoBERTa and GPT2.
Lang	Model	k	τ	Scores
NLI	POS	LM
French	WECHSEL@0	1	1	58.4	85.2	2.5e+5
10	0.1	59.8	86.8	2.0e+5
10	1	58.3	84.4	4.8e+5
50	0.1	57.2	83.6	3.1e+6
50	1	54.0	81.6	1.8e+7
FullRand@0	-	-	46.3	60.6	5.7e+6
CamemBERT	-	-	63.5	93.6	-
German	WECHSEL@0	1	1	55.8	72.7	6e+5
10	0.1	58.9	76.0	4.2e+5
10	1	57.5	75.4	8.3e+6
50	0.1	55.4	75.4	1.0e+7
50	1	53.6	69.5	5.9e+7
FullRand@0	-	-	44.5	49.1	6.2e+6
GBERTBase	-	-	63.2	81.4	-
Chinese	WECHSEL@0	1	1	47.4	75.4	2.7e+6
10	0.1	48.0	80.7	2.6e+6
10	1	48.3	80.3	3.1e+6
50	0.1	48.3	77.8	3.7e+7
50	1	47.9	76.5	8.6e+7
FullRand@0	-	-	37.5	53.7	5.8e+6
BERTBase-Chinese	-	-	61.9	91.9	-
Table 7: Grid search over the temperature  $\tau$  and number of most similar tokens  $k$  parameters of WECHSEL.
We call this variant of our method WECHSEL $_{\mathrm{TFR}}$ . We evaluate WECHSEL $_{\mathrm{TFR}}$  by training the same models as for WECHSEL. Results are shown in Table 9 for GPT2 and in Table 10 for RoBERTa. We find that, on average, performance is on par with WECHSEL.
E Choosing a transfer baseline
We consider two baseline methods to transfer models to a new language without using any language-specific information. One method copies non-embedding parameters to the target language and initializes embeddings from a random normal distribution as done by de Vries and Nissim (2021). We refer to this method as TransInner. Another option copies non-embedding parameters and assigns the embedding of a random token in the source language to each embedding in the target language (effectively "shuffling" the embeddings) as done by Zoph et al. (2016) and Nguyen and Chiang (2017). We refer to this method as TransInnerShuffleEmb. We evaluate these two methods using a setup equivalent to the experiments in Section 5.3 and find that TransInner performs slightly better than TransInnerShuffleEmb (Figure 6), so we use TransInner for subsequent experiments.
Figure 5: WECHSEL, an alternative subword embedding computation method. First, tokenize all words in the word embeddings. Then flatten the result by assigning the embeddings of the words in which it occurred and their word frequencies to each subword. Finally, reduce the embeddings assigned to each subword by taking their mean, weighted by word frequency.
Figure 6: Comparison of German GPT-2 models trained with WECHSEL, TransInner and TransInnerShuffleEmb between freezing non-embedding parameters at the start and not freezing any parameters.
F Sensitivity Analysis w.r.t. training data size
Evaluating on languages with different amounts of available data only indirectly measures the effect of training data size on WECHSEL since other factors (e.g. language similarity to English) are also involved. We conduct a sensitivity analysis to make the relation to the amount of training data explicit (Table 11). Due to computational constraints we only do this for French. We find that the improvement from WECHSEL increases as the amount of training data decreases. In addition, we find that using fastText embeddings trained on less data deteriorates performance, but still leaves a clear margin to TransInner and FullRand.
Lang	Target Token	Closest English Token
French	héritage	legacy
tremp	soaked
étiquoc	bishop
scandaleux	udicrous
vertig	astonishing
enregister	rec
sucrés	sweets
Emmanuel	Emmanuel
entourage	confid
secretariat	ariat
German	machen	ize
mit	with
Sprichwort	proverb
erischen	Austrian
minuten	utes
Haustechnik	umbing
dringen	urgent
verfeinern	refine
umgebung	vironments
ternehmen	irms
Chinese	到处	everywhere
巧合	coinc
第三	third
杂交	recomb
利来	chnology
政务	Govern
石	stone
喊麦	sing
中海	iterranean
张某	defendant
Swahili	shirikishe	ive
Harusi	Marriage
pesile	ery
tihani	graduate
changi	ool
kuugua	ingestion
kuzidi	acclaim
vipigo	Trouble
dhamiri	conscience
aliposimama	Slowly
Table 8: Samples of tokens in each language and the corresponding closest tokens from the English vocabulary according to WECHSEL.
Lang	Model	PPL@0	PPL@25k	PPL@250k
French	WECHSEL-GPT2	1.7e+3	23.47	19.71
WECHSEL_TFR-GPT2	2.3e+3	23.45	19.70
German	WECHSEL-GPT2	3.7e+3	34.35	26.80
WECHSEL_TFR-GPT2	5.0e+3	34.46	26.82
Chinese	WECHSEL-GPT2	2.4e+4	71.02	51.97
WECHSEL_TFR-GPT2	2.5e+4	72.11	52.07
Swahili	WECHSEL-GPT2	1.4e+5	13.02	10.14
WECHSEL_TFR-GPT2	1.5e+5	13.03	10.06
Table 9: Results of training WECHSEL $_{\text{TFR}}$  GPT2 models. We report Perplexity before training  $(\mathbf{PPL}@\mathbf{0})$ , after  $10\%$  of steps  $(\mathbf{PPL}@\mathbf{25k})$  and after training  $(\mathbf{PPL}@\mathbf{250k})$ .
Lang	Model	Score@0	Score@25k	Score@250k
NLI	NER	Avg	NLI	NER	Avg	NLI	NER	Avg
French	WECHSEL-RoBERTa	78.25	86.93	82.59	81.63	90.26	85.95	82.43	90.88	86.65
WECHSEL_TFR-RoBERTa	78.25	87.43	82.84	81.86	90.07	85.96	82.55	90.80	86.68
German	WECHSEL-RoBERTa	75.64	84.53	80.08	81.11	89.05	85.08	81.79	89.72	85.76
WECHSEL_TFR-RoBERTa	77.00	84.70	80.85	80.71	89.09	84.90	82.04	89.72	85.88
Chinese	WECHSEL-RoBERTa	63.23	72.79	68.01	77.19	79.07	78.13	78.32	80.55	79.44
WECHSEL_TFR-RoBERTa	62.75	72.87	67.81	77.07	78.03	77.55	77.99	80.65	79.32
Swahili	WECHSEL-RoBERTa	60.28	74.38	67.33	73.87	87.63	80.75	75.05	87.39	81.22
WECHSEL_TFR-RoBERTa	60.14	75.42	67.78	74.04	87.79	80.92	74.58	87.66	81.12
Table 10: Results from fine-tuning WECHSEL  ${}_{\mathrm{{TFR}}}$  -RoBERTa models. Results shown equivalently as in Table 2.
Model	Subsample Size	Best PPL
16MiB	64MiB	256MiB	1024MiB
WECHSEL-GPT2 (original fastText embeddings)	78.33	44.75	31.63	24.66
WECHSEL-GPT2 (fastText embeddings trained on subsample)	97.42	49.50	32.88	24.75
FullRand-GPT2	281.46	83.43	43.08	27.09
TransInner-GPT2	216.37	77.71	35.27	25.15
Table 11: Sensitivity Analysis w.r.t. the amount of training data on transfer to French. We train models on random subsamples of 16MiB, 64MiB, 256MiB and 1024MiB of the original training data, and evaluate on the same held-out set. For WECHSEL-GPT2, we train two models. One using the original, publicly available fastText embeddings trained on Common Crawl data. The other using fastText embeddings trained only on the corresponding subsample of text.