SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we try to push our most difficult case of ennews-zhwiki further by varying hyperparameters as well as dimensions. In our main results in Figure 3, ennews-zhwiki barely had any transition before reaching high accuracy at 800MB, making us suspect there might be signal before that but the hyperparameter was not optimal. So here we try clip at  $1.5\%$  for coopmap-clip-1.5, drop-1.5 instead of  $1\%$  as in every other experiment. We also adjust drop to  $\min(20, 20\frac{d}{400})$  instead of a constant 20 so that less is dropped for lower dimensions  $d$ .
In Figure 7, we get the typical behavior of increasing accuracy with increasing dimension for 1000MB of data all the way to the end. For 100MB and 300MB, this only worked when we also truncate some of the tail end of the SVD. Even here, the maximum accuracy was reached at around 2000 dimensions out of 5000, reaching  $>50\%$  accuracy at 300MB of data and 2000 dimensions. It is also remarkable that while clipping more resulted in more stability, the final accuracies are virtually identical when both clip and clip-1.5 works or when both drop and drop-1.5 works. This is a case where it is helpful to truncate some of the tail of the SVD as well, at least for 100MB and 300MB where all failed at full dimensions. However, the accuracy still increased up to 2000 dimensions, much higher than typical vectors for the vocabulary size according to conventional wisdom.
D More details and ablations
D.1 Effect of initialization
Figure 8: Accuracy vs. dimensions on enwiki-esnews. 1000MB of data. Left: use full coocmap initialization. Top red line is when coocmap-drop uses all 5000 dimensions. Right: use their own initialization.
To rule out that fasttext is only limited by the unsupervised initialization step, Figure 8 shows when all methods use the unsupervised initialization of full rank coocmap (left), vs their own unsupervised initialization (right). Here vecmap-fasttext and coocmap-fasttext-sqrt both worked for a slightly wider range of dimensions but the differences are small. Using the clipped version of initialization also did not make much difference. Overall, varying initializations did not matter too much for unsupervised initializations. On the other hand, for supervised initialization, coocmap tend not be very sensitive once there is enough data. vecmap can be more sticky and is more affected by supervised initialization, possibly because  $W$  can reach a local minimum.
D.2 Effect of matching method
The matching method was critical in getting coocmap to work, where simply applying the bidirectional matching does not work even on the simplest enwiki-eswiki. It starts to work in the first few iterations,
but then half of the vocabulary gets matched to the one word (often but not always [UNK]) and accuracy go to 0. Some hubness mitigation (Dinu et al., 2015) is a must for coocmap. This is not surprising since the indices  $s, t$  can make arbitrary matches, whereas the vector space cannot make arbitrary matches from a rigid rotation.
We tried a naive and greedy mutual matching method that also solved the problem, but CSLS is better, simpler and is more similar to prior works. A proper matching methods such as Hungarian algorithm or softer matching method already tested on bilingual dictionary induction (Marchisio et al., 2022a) should work as well.
Subwords. we used fasttext since it is more standard for this problem, and we also checked that the subwords information made no difference by turning them off in fasttext hyperparameters. This is reassuring for the findings to apply to word2vec as well.
E From vectors to association matrices
We derive a more precise relationship between coocmap and vecmap using the notations of Section 3.
Let the  $d$ -dimensional vectors be  $X, Z \in \mathbb{R}^{|V| \times d}$  such that they are whitened  $X^{\intercal} X = Z^{\intercal} Z = I_d$ . Then for coocmap, the association matrices is  $K(Y) = (YY^{\intercal})^{1/2} = YY^{\intercal}$ . We can show that for dot product cdist, and permutations  $s, t$ ,
$$
\operatorname {c d i s t} (K (X) [:, s ], K (Z) [:, t ]) = \operatorname {c d i s t} (X W, Z) \tag {5}
$$
where the LHS is the distance function in coocmap of Alg 1 and the RHS is the distance function in vecmap of Alg 2. To see this,
$$
\begin{array}{l} \operatorname {c d i s t} (K (X) [:, s ], K (Z) [:, t ]) \\ = K (X) [:, s ] \cdot K (Z) [:, t ] ^ {\intercal} \\ = (X X [ s ]) ^ {\intercal} \cdot (Z Z [ t ] ^ {\intercal}) ^ {\intercal} \\ = X X [ s ] ^ {\intercal} \cdot Z [ t ] Z ^ {\intercal} \\ = X (X [ s ] ^ {\intercal} Z [ t ]) Z ^ {\intercal} \quad \text {i f} X [ s ] W = Z [ t ], \text {t h e n} W = (X [ s ] ^ {\intercal} X [ s ]) ^ {- 1} X [ s ] ^ {\intercal} Z [ t ] \\ = X W Z ^ {\intercal} \quad \text {s i n c e} (X [ s ] ^ {\intercal} X [ s ]) ^ {- 1} = I, W = X [ s ] ^ {\intercal} Z [ t ] \\ = \operatorname {c d i s t} (X W, Z) \\ \end{array}
$$
We can try relaxing the whitening assumption. w.l.o.g., let  $X = U_{1}\Sigma_{1}$ ,  $Z = U_{2}\Sigma_{2}$  where  $U_{i}^{\mathsf{T}}U_{i} = I_{d}$ . This can be obtained via SVD on any  $X' = U_{1}\Sigma_{1}V_{1}^{\mathsf{T}}$  so that the association matrix is not affected, i.e.  $K(X') = (X'X'\mathsf{T})^{1/2} = (XX')^{1/2} = U_{1}\Sigma_{1}U_{1}^{\mathsf{T}}$ . In this case,  $W = U_{1}[s]^{\mathsf{T}}U_{2}[t]$  is the mapping that would make the two sides equal.
However, if we try least square solve as before, we get  $W = \Sigma_1^{-1} U_1[s]^T U_2[t]\Sigma_2$ . The Procrustes solution for  $X[s]W = Z[t]$  (or  $U_1[s]\Sigma_1W = U_2[t]\Sigma_2$ ) is  $W = \mathrm{svd - norm}(\Sigma_1^\top U_1[s]^\top U_2[t]\Sigma_2)$ . svd-norm takes the SVD but set all singular values to 1. Neither will make the two sides equal exactly but these have some similarities.
F What is in higher dimensions?
Although the increasing accuracy, the absolute results and the consistency when we keep higher dimensions suggest that there are useful signals, it could always be that we are using lower dimensions poorly or benefiting from other unknown effects. In this section, we compare the full-rank matrix and the low-rank matrix to see what exactly is the difference. If the information lost by low rank seems useful or robust, then we can be more confident in retaining this information. Indeed, the low rank version seem to lose or de-emphasize a lot of world knowledge, specific phrases and the like while making the association matrix more like a similarity matrix. In this section, we compare the full-rank matrix and the low-rank matrix to see what exactly is the difference. If the information lost by low rank looks useful or robust, then we can be more confident in retaining this information. Indeed, the low rank version seem to lose or de-emphasize a lot of world knowledge, specific phrases and the like while making the association matrix more like a similarity matrix.
We construct the full dimensional matrix, apply normalize and drop. Then we use the thresholds established by clip. If an entry needs to be clipped in the full-rank matrix, but somehow is already
reduced and no longer need to be clipped in the lower-rank matrix, we record this as full-rank+ in Table 9. Conversely, if low-rank instead increased an entry so that it should be clipped, we note it as 300-dim+.
From these examples, the differences are clear. The full dimensional matrix contains more world knowledge and otherwise favor dissimilar words that are highly related: richard-nixon (famous person), portland-oregon maine (geography), error-margin, molecular-weight (scientific term), tokyo-1964 (olympic), basketball-jordan james (famous players), cbs-60 minutes (show on CBS), louisiana-parish purchase (famous event).
In contrast, the  $300\text{-dim}+$  favors identical or similar words like state-state, age-age, who-whom, truth-true, tokyo-pyeongchang, cbs-fox, family-households, making the association matrix more like the similarity matrix. This is also intuitive from SVD as well: if the association matrix is  $USV^{\intercal}$ , then the similarity matrix is  $US^{2}V^{\intercal}$  whereas low dimension is  $US_{d}V^{\intercal}$  with both  $S^{2}$  and  $S_{d} = \mathrm{trunc}_{d}(S)$  emphasizing higher singular values.
We also tested full-rank vs 1000-dim, where the effects are similar to the full-rank vs 300-dim shown, but more subtle and harder to present clearly. When we tested drop we found drop tend to reduce associations between pairs of numbers, words and punctuations and the like, whereas it increased associations between article and words. It is not clear if this is good or bad.
G Example predictions
We include some predictions for extra information. The  $\mathrm{P@1}$  is quite generous to small vocabulary  $V$ , since it does not evaluate on entries if the source is not in  $V_{1}$ , or if no targets are in the  $V_{2}$ , treating the evaluation as out-of-vocabulary instead. So we show the number of correct predictions and the source overlap with the reference dictionary, both out of 5000. Results are shown in Table 10.
source	full-rank+	source	full-rank+
NewsCrawl	Wikipedia
richard	nixon	families	residing
cooper	anderson	called	so
anderson	cooper	aspect	ratio
report	contributed	error	margin
at	least	walter	scott
no	longer	yet	another
than	rather	oregon	portland
this	earlier	scott	walter
dean	wells	unless	otherwise
kyle	walker	usa	today
crazy	rich	jones	indiana
behind	scenes	louisiana	parish
walker	kyle	british	columbia
politically	correct	water	%,
thomas	cook	cooper	alice
graham	billy	maryland	baltimore
sense	common	why	reason
detroit	lions	illinois	chicago
correct	politically	lord	rings
aaron	rogers	james	bond
2000	russell	siege	laid
star	wars	%.	persons
light	shed	molecular	weight
town	hall	then	since
almost	certainly	bob	hope
hollywood	reporter	if	even
texas	austin	coast	guard
cash	flow	final	fantasy
full-rank+	300-dim+
source: tokyo
NewsCrawl
japanese	winter
prosecutors	south
electric	china
metropolitan	korea
detention	pyeongchang
Wikipedia
bay	chinese
1964	china
2020	korea
ward	areas
stock	san
rose	taiwan
source: cbs
NewsCrawl
face	fox
60	night
nation	channel
boston	telling
minutes	abc
Wikipedia
60	bbc
minutes	disney
columbia	cable
walter	live
rather	shows
full-rank+	300-dim+
source: truth
NewsCrawl
speak	true_
commission	are
moment	than
post	be
uncomfortable	story
Wikipedia
tell	god
commission	view
table	than
functional	universe
search	which
source: basketball
NewsCrawl
operations	club
corruption	rugby
division	nfl
courts	clubs
magic	golf
james	ball
Wikipedia
nba	club
operations	stadium
country	ice
memorial	rugby
college	clubs
Figure 9: Left: word pairs favored by the full rank matrix (full-rank+) on NewsCrawl or Wikipedia sorted by the amount of difference over all pairs of words. 300-dim+ not shown since 72/top 100 consists of identical words with the rest mostly synonyms. Right: same comparison but for selected words, also sorted by the difference.
ennews	zhwiki	enwiki	esnews	enwiki	dewiki
4458 swift	〉	3484 victims	víctimas	2732 looking	ergaben
947 significant	重大	2727 somewhat	bastante	3056 deputy	stellvertreter
2179 strength	力量	682 star	estrella	1805 1977	1977
2972 letters	信	2847 centers	centros	337 september	september
2401 coverage	直播	4176 theology	sindicatos	4669 strict	streng
733 idea	提议	868 historic	históricos	2248 sector	bereich
4260 assist	协助	451 research	investigaciones	414 came	kam
2803 properties	特性	3548 tell	comprobar	105 while	hingegen
4542 circuit	法官	2375 script	estuvieron	86 county	county
4191 saint	圣	3570 elevation	mínima	702 pacific	bewohnern
612 recently	近年	3021 800	600	248 house	haus
3360 maria	娜	765 forms	formas	1178 product	produkt
1846 surprise	在内	4575 delayed	adelantado	164 second	zweite
713 copyright &	&	731 r	r	1907 contained	enthielt
2015 carrying	携带	4058 accurate	precisa	3208 easy	schwer
3843 suspects	自杀	3708 technologies	technologías	3200 tone	ton
1536 dropped	下降	854 elements	elementos	4578 consensus	zustande
1014 largest	最大	3187 abbey	estás	477 television	fernsehen
2672通讯	4934 masters	公开赛	1483 mountains	sierra	1716 75	75
989 words	词	4708 southwestern	básico	4489 handed	übergeben
4400 pan	亚	1094 problems	problemas	2866 electrical	elektrische
3289 maximum	以上	3813 compete	competir	2976 proposal	vorschlag
1620 ceo	总裁	934 remains	sido	4788 beta	)-
821 offer	提供	1642 crown	corona	691 food	nahrung
2611 150	150	3045 harbor	sentirse	251 power	macht
1520 individual	个人	4274 phone	teléfono	725 able	konnte
801 cancer	疾病	3626 z	subraya	286 late	späten
4003 ."	任内	302 german	alemán	290 park	park
319 always	总是	133 early	naranja	3903 en	en
4424 backs	司职	4955 provisions	vigente	808 source	quelle
1345 modern	现代	3366 fruit	aceite	1050 cost	kosten
4835 spell	词	2249 engines	motores	1803 99	99
4737 loud	声	4508 trump	trump	2179 stand	stehen
3894 metres	公尺	2669 forming	formar	3926 pitch	ton
4472 flood	洪水	764 personal	personales	4449 populated	besiedelt
4648 attract	吸引	1810 identity	identidad	3979 hunt	jagd
3297 lowest	最低	782 strong	fuerte	4909 wore	trugen
539 50	50	936 recent	recientos	2510 seems	scheint
1083 build	建造	955 always	siempre	3978 cousin	bruder
72 into	nos	2325 operate	beteiben
correct 1411 source 3341 P@1 66.30%	correct source P@1 71.60%	correct source P@1 77.05%
Figure 10: 40 random samples from all predictions. Source indices are included to reflect frequency where smaller means more frequent in the source language (left columns).