SlowGuess/NAACL_2022_OCR huggingface dataset:
We collect example sentences containing the evaluation words for each language from 2018 Wikipedia dumps<sup>14</sup>. We use the Perl script<sup>15</sup> from linguatools to convert xml format to raw text, excluding paragraph and heading mark-ups, and math and table tags. From raw text, we collect context sentences containing the evaluation words. Due to relatively insignificant differences between using 10 or 100 context examples for embedding extraction (Vulić et al., 2020b), we use 10 context examples for speed in running experiments. We choose the  $L \leq 8$  layer setting and all other optimal settings from the original paper (Vulić et al., 2020b). We find pretrained BERT models for all languages except FR, for which we use a similar model called FlauBERT (Le et al., 2020). Pretrained models used in our re-implementation are given in Table 4. Note that the FR and RU models are cased, which may slightly affect the results. These were the only models we could find for these languages.
A.2 OOV Words
A detailed breakdown of OOV words by method is given in Table 5.
A.3 Heatmaps per Language
Fig. 4 shows heatmaps for languages missing from Fig. 2 for COLEXmono, COLEXcross, C+F Fusion<sup>16</sup>.
$^{14}$ https://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/ $^{15}$ https://www.dropbox.com/s/p3ta9spzffviovk0/xml2txt.pl?dl=0 $^{16}$ The discussion of Fig. 2 is focused around low-resource languages.
AR EN ES FI FR HE PL RU ZH
https://huggingface.co/asafaya/bert-base-arabic  
https://huggingface.co/bert-base-uncased  
https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased  
https://huggingface.co/TurkuNLP/bert-base-finnish-uncased-v1  
https://huggingface.co/flaubert/flaubert_basecased  
https://huggingface.co/onlplab/alephbert-base  
https://huggingface.co/dkleczek/bert-base-polish-uncased-v1  
https://huggingface.co/DeepPavlov/rubert-base-cased  
https://huggingface.co/bert-base-chinese
Table 4: Links to pretrained BERT models for each language
AR	EN	ES	FI	FR	HE	PL	RU	ZH
fastText	0	0	0	0	0	0	0	0	0
BERT	0	0	0	0	0	0	0	0	0
BabelNet	709	0	84	72	33	374	342	479	209
ARES	824	10	99	84	47	469	372	628	250
COLEXmono	860	45	117	140	68	623	547	710	295
COLEXmaxsim	755	0	88	73	34	402	350	526	226
COLEXcross	755	0	88	73	34	402	350	526	226
Table 5: OOV words from Multi-SimLex for each approach. We provide OOV words for each language when querying BabelNet, because all COLEX approaches and ARES rely on BabelNet synset annotations. Any further OOV words for COLEX and ARES approaches beyond those not in BabelNet are due to not having at least one synset embedding for an evaluation word. For ARES, we are restricted to the pretrained embeddings provided at http://sensembert.org/resources/ares_embedding.tar.gz. For COLEX, a synset must be colexified at least once to have a vector representation.
EN	ES	FI	FR	HE	PL	RU	ZH
EN	-	3222/3318	3275/3352	2257/284	2794/3274	2798/3274	2551/3222	2913/3151
ES	3222/3318	-	3084/3256	2544/2645	2704/3256	2681/3250	2428/3189	2805/3116
FI	3275/3352	3084/3256	-	2595/2682	2718/3243	2756/3294	2502/3257	2850/3137
FR	2257/2284	2544/2645	2595/2682	-	2462/2903	1972/2379	1696/2219	2041/2243
HE	2794/3274	2704/3256	2718/3243	2462/2903	-	2391/3201	2242/3226	2453/3056
PL	2798/3274	2681/3250	2756/3294	1972/2379	2391/3201	-	2262/3209	2419/3009
RU	2551/3222	2428/3189	2502/3257	1696/2219	2242/3226	2262/3209	-	2244/3032
ZH	2913/3151	2805/3116	2850/3137	2041/2243	2453/3056	2419/3009	2244/3032	-
Table 6: Ratio of cross-lingual word pairs used for each language pair. Numerator represents number of word pairs used and denominator represents total word pairs provided in Multi-SimLex for each language pair.
(a) COLEXmono (ES)
(b) COLEXcross (ES)
(c) C+F Fusion (ES)
(d) COLEXmono (FR)
(e) COLEXcross (FR)
(f) C+F Fusion (FR)
(g) COLEXmono (ZH)
(h) COLEXcross (ZH)
(i) C+F Fusion (ZH)
Figure 4: Heatmaps for ES, FR, and ZH. See Fig. 2 from main paper.