SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the small but clean Europarl data, we tested on English to Finnish (fi) and Hungarian (hu). As shown in Figure 2, coocmap started to work at 9MB for hu and 30MB for fi. It finished the transition region when we used all 300MB of en-fi and 100MB of en-hu. vecmap-fasttext has yet to work, agreeing with the results of Søgaard et al. (2018). However, since vecmap-raw using simple SVD worked, it was surprising that fasttext did not. Indeed, decreasing the dimension to 100 from 300 enables vecmap-fasttext to start working at 50MB of data for enparl-huparl (vs 8MB for coocmap, Figure 6).
Next result is on English to Chinese (zh), where both are trained on Wikipedia. coocmap had a bit more instability vecmap-fasttext also sometimes works with around 1GB of data. The more robust version of coocmap-clip and drop is completely stable, and begin to work with less than 20MB of data.
4.2 Under domain mismatch.
The most difficult case for unsupervised word translation is when data from different domains are used for each language (Søgaard et al., 2018; Marchisio et al., 2020). The original vecmap of Artetxe et al. (2018b) was tested on different types of crawls (WacKy, NewsCrawl, Common Crawl) which did not work in previous methods. Marchisio et al. (2022b) show that vecmap also failed for NewsCrawl to Common Crawl on harder languages. We test on Wikipedia to NewsCrawl and Europarl and see successes on enwiki-esnews, enwiki-hunews, and ennews-zhwiki, enparl-eswiki before finally failing on enwiki-fiparl and enwiki-huparl. See Figure 3.
On enwiki-esnews, where  $\sim 100\mathrm{MB}$  of data was required to reach  $50\%$ , though the basic coocmap becomes unstable and all vecmap or fasttext based methods failed to work at all. However, clip and drop not only stabilizes this but enabled it start to work at 40MB of data.
On enwiki-hunews, coocmap mostly fails but get  $5\%$  accuracy with above 100MB of data. Impressively, clip and drop fully solves this problem as well, but even clip has a bit of instability, reaching  $50\%$  accuracy at 600MB of data.
On ennews-zhwiiki, coocmap fails completely without clip and drop, and even then requires more data than before. Finally it works and reaching over  $50\%$  accuracy around 800MB. In C, we show the data still has higher potential, where truncating to 2000 dimensions enables ennews-zhwiiki to work with 300MB or even 100MB of data, though still not reliably.
For more extreme domain mismatch, we test enparl-eswiki. In addition to being small, Europarl contains only parliamentary proceedings which has a distinct style and limited range of topics, to our surprise this also worked with 295MB of enparl, and 500MB from eswiki, reaching a final accuracy of  $70\%$  suddenly, although basic coocmap also showed no sign of working. All methods failed for enwiki-fiparl and enwiki-huparl in the range limited by Europarl (300MB for fiparl and 90MB for huparl) with up to 1GB of enwiki, reaching the end of our testing.
5 Analysis: why coocmap outperformed dense vectors
These main results show that high-dimensional coocmap is more data efficient and significantly more robust than the popular low-dimensional word vectors fasttext/word2vec, which contract prevailing assumptions that vectors are superior and necessary to enable unsupervised translation among other tasks. Ruder et al. (2019) states "word embeddings enables" various interesting cross-lingual phenomena. For unsupervised dictionary induction, landmark papers (Lample et al., 2018a; Artetxe et al., 2018b) needed vectors and even methods that must use a  $|V| \times |V|$  input constructed these from low dimensional vectors anyway (Alvarez-Melis and Jaakkola, 2018; Marchisio et al., 2022a). More generally, the popular textbook Jurafsky and Martin (2023, 6.8) states "dense vectors work better in every NLP task than sparse vectors". Here, we provide natural reasons that disadvantage vectors if we do not mind having higher dimensions.
The conflicting dimensions of vectors. These properties must hold for vectors to work,
1. Approximate isomorphism: be able to translate by rotation/linear map
2. Denoise: reduce noise and information not robust for the task
3. Retention: keep enough information at a given dimension to achieve good performance
By testing the dimension of word vectors in Figure 4, we can see that isomorphism and denoising only holds in low-dimensions. To see this on enwiki-dewiki where fasttext works well, both vecmap-
Figure 3: Accuracy vs. data size with clip and drop. Except for enwiki-eswiki and enwiki-zhwiki (top left, top middle), the rest all have domain mismatch where vecmap-fasttext gets  $\approx 0$ .
Figure 4: Accuracy vs. dimension at 300MB of data. coocmap-clip/drop:  $\mathrm{trunc}_d(X)$  by svd, coocmap-fasttext:  $X =$  normalize $(X'X'\mathsf{T})^{\frac{1}{2}})$  for  $X' \in \mathbb{R}^{|V| \times d}$ , vecmap-fasttext:  $X' \in \mathbb{R}^{|V| \times d}$ .
fasttext and coocmap-fasttext are better than SVD truncation at low dimensions. Then accuracy goes to 0 as dimension increases. This is not because vectors fail to retain information, since coocmap-fasttext-clip works if we apply additional clipping. See B for more testing on the effect of dimension and other hyperparameters. Notably, lower dimensions have better isomorphism and denoising, and higher dimensions have better accuracy and retention. This trade off leaves a limited window for fasttext to work well. Still, selecting a good dimension is sufficient for fasttext vectors to match the accuracy and data efficiency of coocmap on the easiest cases (en-es, fr, de) and work well enough on dissimilar languages training on the same domain.
Søgaard et al. (2018) notes similar impact of dimensionality themselves, but without exploring enough range that would have solved their enparl-huparl and enparl-fiparl tests (with vecmap). Yin and Shen (2018) argues that optimal dimension may exist because of bias-variance tradeoff. Our experiments show that coocmap keeps improving with higher dimensions, and they may have been misled by relying on linear algebraic properties. The unreliability of isomorphism is also noted by Patra et al. (2019); Marchisio et al. (2022b).
Better denoising in high dimensions. Domain mismatch is where the vectors struggle the most regardless of dimension. In the easiest domain mismatch case of enwiki-esnews of Figure 4, vecmap-fasttext failed whereas coocmap-fasttext-clip worked (though not stably), showing that clip helps even when applied on top of the natural but incidental denoising of vectors.
The association matrix of coocmap normalize  $(\mathrm{Co}^{\circ 1 / 2})$  is also better overall than other choices such as Mikolov et al. (2013a); Levy and Goldberg (2014); Pennington et al. (2014); Rapp (1995). In A, we compared to other full-rank association matrices corresponding to popular vectors and show they also perform worse than coocmap. For instance, the positive pointwise mutual information matrix (PPMI) of Levy and Goldberg (2014) corresponds to word2vec/fasttext. While it can work with
simple  $\ell_2$  cdist without normalize and can reach higher accuracy than the basic coocmap if tested on the same domain, it has lower data efficiency and completely fails on the easiest enwiki-esnews with domain mismatch, showing the same behaviors as fasttext. A more aggressive squishing such as normalize  $(\log (1 + \mathrm{Co}))$  works robustly too but is less accurate and less data efficient. Impressively, Rapp (1995) also works fully unsupervised with the prescribed  $\ell_1$ .
Higher dimensions contain useful information. In all cases, coocmap-clip and coocmap-drop both become more accurate as dimension increases. Under domain mismatch, there is considerable gains above even 1000 dimensions, suggesting that we are losing useful information when going to lower dimensions. Analyzing the association matrices based on which values are clipped, we show in F that higher dimensions retain more world knowledge such as portland-oregon, cbs-60 minutes, molecular-weight, basketball-jordan, luisiana-purchase, tokyo-1964 and using low dimensional vectors misses out.
6 Discussions
Better vectors exist. coocmap shows that it is feasible to denoise in high dimensions better than with popular vectors. Thus, applying denoising suitable to task, the size/type of training data first before reducing dimension is a more principled way to train lower dimensional vectors than the current practice of relying on poorly optimizing a supposed objective. In this view, training vectors can just focus on how to best retain information in low dimensions and vectors should just get monotonously better with increasing dimensions and stop at an acceptable loss of information for the task and data.
There is room for improvements for vectors of all dimensions. In Figure 4, fasttext got higher accuracy at impressively low dimensions in favorable conditions whereas SVD on coocmap-drop becomes more accurate with increasing dimension. Without actually constructing the better vectors, it is clear that meaningful improvements are possible: for any given dimension  $d$ , there exists rank- $d$  matrices that can achieve a higher accuracy than the max of all rank- $d$  methods already tested in Figure 4.
But why vectors? Here is our opinion on why low-dimensional vectors appeared necessary and better beside social reasons. Co-occurrence counts from real data are noisy in very involved ways and violate standard statistical assumptions, however it is obligatory to apply principled statistics such as likelihood based on multinomial or  $\chi^2$ , whereas vectors take away these bad options and allow less principled but better methods. Statistics says likelihood ratios is the most data efficient:  $D(s,t) = \sum_{i}p(s,i)\log \frac{p(s,i)}{p(t,i)}$ , or perhaps the Hellinger distance  $H(s,t) = \sum_{i}||p(s,i)^{\frac{1}{2}} - p(t,i)^{\frac{1}{2}}||_2^2$  if we want more robustness. Using raw counts or a term like  $p\log p$  (super-linear in raw counts) always failed when measuring with the  $\ell_2$  norm. The likelihood-based method of Ravi and Knight (2011) likely would fail on real data for these reasons. Normalization is a common and crucial element of all working methods. The methods of Rapp (1995); Fung (1997); Rapp (1999) were based on statistical principles but with modifications to make normalized measurements. Levy and Goldberg (2014); Pennington et al. (2014) actually proposed better normalization methods too that would equally apply to co-occurrences (Appendix A).
Giving up on statistics and starting from vecmap lead to big improvements, ironic for our attempt to understand unsupervised translation and how exactly vectors enabled it. To prefer more empirical tests on a wider range of operations instead of believing that we can model natural language data with simple statistics might be another underrated lesson from deep learning. In this work, we give up on statistical modelling without giving up on higher dimensions, which makes available these key tools that were sufficient to outperform low-dimensional vectors: large squeezing operations like sqrt, normalizing against a baseline (subtracting means, divide by marginals), contrasting with others during matching (CSLS), clip and drop. With just normalize and CSLS, coocmap would have similar accuracy and higher data efficiency compared to fasttext. Clip made co-occurrences more robust than vectors while also increasing accuracy, showing the possibility of denoising in high-dimensions. Finally, drop enabled some access to the information in higher dimensions and lead to noticeably higher accuracy as dimension increases. This is actually a more intuitive situation than the prevailing understanding that going to very low dimensions is both more compact and more accurate.
IBM models. coocmap shows that unsupervised word translation would have been possible with the amount of compute and data used to train IBM word alignment models (Brown et al., 1993).
While we do not test on the exact en-fr Hansards (LDC), we know it has  $>1.3$  million sentence pairs from the Parliament of Canada. This can be compared to our Europarl en-hu experiments, which has 0.6 million sentence pairs for a more distant language pair, showing there may not be much additional information from the sentence alignments. coocmap with drop got around  $75\%$  accuracy here. Though the compute requirement of coocmap may have been difficult in '93 – by our estimate en-hu with  $|V| = 5000$  probably requires  $|V|^3 * 100 \approx 1$  Tera-FLOs, which should have been easy by the early 2000s on computers having Giga-FLOPS.
Limitations. The vocabulary size is a potentially important parameter unexplored in this paper where we used a low 5000 for cleaner/faster experiments. The most expensive step is factorization operations and measuring distances of high dimensional co-occurrences which is  $O(|V|^3)$  as opposed to  $O(d|V|^2)$  for vectors, though approximations are available with dimension reduction (see E). Having a low vocabulary size may explain why we see little benefit (except in our hardest cases such as C) from the potential denoising effects of truncation. Low-dimensional vectors is more easily scalable to larger vocabulary and can more easily be used by neural models. However, BPE, indexing, truncating vocabulary, multiple window sizes and other alternatives are not explored to better use higher dimensions.
Speculations. As cross-domain is more challenging than just cross-lingual, it is especially likely that our findings may apply in other cross-domain tasks. So we speculate that if co-occurrences are similarly processed without going to low dimensions, they are likely to perform better on other tasks where more robustness is required than the natural robustness of low-dimensions. Similarly, neural models may also benefit from intentional denoising like clip and drop.
Beyond simple vectors, more recent language models may also suffer from low dimensions, which might be why they benefit from retrieval (Khandelwal et al., 2020), where low dimensional vectors may lose too much information for the task. Recently, contextual transformer representations superseded non-contextual vectors with more data and more dimensions pushing up the performance across tasks. Following Kaplan et al. (2020), there are important trade-off between dimensions, data and compute. If data is the limiting factor, it is plausible that better ways of using higher dimensions is helpful.
Related work. Rapp (1995) already hinted that there might be enough signal for fully unsupervised translation. Fung (1997); Rapp (1999); Haghighi et al. (2008) tested their high dimensional approaches with seed words, but without a full search procedure. Ravi and Knight (2011) succeeded in the case of time expressions and parallel data, but likely used insufficient normalization for harder cases. Mikolov et al. (2013b) noticed the relation between low-dimensional rotations and translation. With good enough vectors, Zhang et al. (2017a); Lample et al. (2018a); Artetxe et al. (2018a) achieved fully unsupervised word translation while showing that seed words do not matter too much as long as the unsupervised method works. There was much follow up on these successes, in particular, even methods that must use an association matrix (Alvarez-Melis and Jaakkola, 2018; Marchisio et al., 2022b) constructed them from low-dimensional vectors.
Broader Impact
For dictionary induction, this work shows it can be done with less data and is more robust to domain mismatch than previously thought. We have a working procedure that is more direct, using less compute, making it more accessible for educational purposes. Unsupervised translation means learning from the statistics of data alone and can make bland and silly mistakes where outputs should not be relied on unless they can be verified.
Thanks to Mikel Artetxe, Kelly Marchisio, Luke Zettlemoyer, Scott Yih, Freda Shi, Hila Gonen and Tianyi Zhang for helpful discussions.
David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-wasserstein alignment of word embedding spaces. In Proc. of EMNLP, pages 1881-1890.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In ICLR.
Kelly Marchisio, Neha Verma, Kevin Duh, and Philipp Koehn. 2022b. IsoVec: Controlling the relative isomorphism of word embedding spaces. In Proc of EMNLP, pages 6019-6033.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013c. Distributed representations of words and phrases and their compositionality. In Proc. of Neurips.
Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-top: Simple and effective postprocessing for word representations. In ICLR.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In Proc. of ACL.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proc of EMNLP.
Reinhard Rapp. 1995. Identifying word translations in non-parallel texts. In Proceedings of the Association for Computational Linguistics.
Reinhard Rapp. 1999. Automatic identification of word translations from unrelated English and German corpora. In Proc. of ACL.
Sujith Ravi and Kevin Knight. 2011. Deciphering foreign language. In Proc of ACL.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569-631.
Anders Søgaard. 2023. Grounding the vector space of an octopus: Word meaning from raw text. *Minds & Machines*, 33:33-54.
Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018. On the limitations of unsupervised bilingual dictionary induction. In Proc. of ACL.
Karl Stratos, Michael Collins, and Daniel Hsu. 2015. Model-based word embeddings from decompositions of count matrices. In Proc. of ACL.
Zi Yin and Yuanyuan Shen. 2018. On the dimensionality of word embedding. In Neurips.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017a. Adversarial training for unsupervised bilingual lexicon induction. In Proc. of ACL.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017b. Earth mover's distance minimization for unsupervised bilingual lexicon induction. In Proc. of EMNLP.
A Comparison of association matrices
Figure 5: top: methods from the 90s, bottom: methods from 2010s word2vec and glove.
method	association formula	normalize	cdist
coocmap	Co^o1/2	normalize	l2
log1p	log(1 + Co)	normalize	l2
Rapp (1995)	p(s,i)/p(s)p(i)	l1	l1
Fung (1997)	p(s,i) log p(s,i)/p(s)p(i)	l1	l1
Levy and Goldberg (2014)	max{0, log p(s,i)/p(s)p(i) - log k}	l2	l2
/ Church and Hanks (1990)
Pennington et al. (2014)	log(1 + Co) - b·1T - 1·bT	l2	l2
In Figure 5, we compare to several choices of association method  $K(\mathrm{Co})$  and find all of them to be worse than  $\mathrm{Co}^{\circ 1/2}$  plus normalize overall, and the two popular methods from 2010s are better than the two methods from 1990s. These methods all perform normalization by factoring out the marginals one way or another.
Besides the association matrices themselves, we match them with their best normalization method from normalize,  $\ell_1,\ell_2$ . We report  $\ell_{2}$  rather than normalize for Levy and Goldberg (2014); Pennington et al. (2014) which proposed their own normalization. In all these cases, normalize gave almost identical results as  $\ell_{2}$ , showing their own normalization was sufficient. normalize corresponds to  $\ell_{2}$  cdist since it did  $\ell_{2}$  normalization as the last step in (3). We use  $p(s,i) = \mathrm{Co}(s,i) / \sum_{s^{\prime},i^{\prime}}\mathrm{Co}(s^{\prime},i^{\prime})$  when a probability was required.
On the top of Figure 5, we compare Rapp (1995); Fung (1997) from the 90s specifically designed for dictionary induction as well. Impressively, Rapp (1995) also performed well using  $\ell_1$  distance as prescribed, where it even started to work on enwiki-esnews. Fung (1997) did not work with the prescribed  $\ell_2$  criteria (not shown) but worked with  $\ell_1$  as well. We think Rapp (1999) is similar to our modification of Fung (1997) with  $\ell_1$ , which contains its most important term. However, using  $\ell_1$  leads to lower data efficiency and lower accuracy in methods that also work with  $\ell_2$ .
On bottom of Figure 5, we compare two popular association matrices from the 2010s. Notably, the positive PMI of (Levy and Goldberg, 2014) corresponds to word2vec/fasttext, and actually has similar performance characteristics as fasttext, where both reach higher accuracy than coocmap on easy cases but has lower data efficiency and fails on enwiki-esnews. Other methods that use normalize on log also worked for enwiki-esnews. Of these, coocmap has the highest accuracy. Levy and Goldberg (2014) achieved higher accuracy while Pennington et al. (2014) worked on enwiki-esnews. The normalization proposed in both methods achieved almost identical results with  $\ell_2$  as with normalize.  $\ell_1$  also worked on them but clearly worse and has less data efficiency (not shown).
Pennington et al. (2014) did not specify what the biases  $b$  should be, instead just leaving them to gradient optimization, we picked them to subtract out the mean of logs whereas PMI would subtract the log of means.  $\mathrm{Co}^{\circ 1 / 2}$  was also tested by Stratos et al. (2015) and found to be favourable. We also did not find the shifted PPMI to be better than the basic PPMI of Church and Hanks (1990).
B Improving fasttext
Figure 6: Improving fasttext with hyperparameter and clipping. Drop was set to  $\min(20, 20\frac{d}{400})$  resulting in a double curve since  $d = 100, 300$  were both used.
In the main results, we used default parameters, where the important ones were skigram, lr: 0.05, dim: 300, epoch: 5. In analysis section, we improve fasttext further by tuning dimension, learning rate and epoch number, then by using coocmap, clip and drop. The effect of dimension has been specifically explored in Figure 4. Here we show how 100 dimension and 300 dimensions as we vary with the size of data - 100-dim tend to be more data efficient whereas 300-dim tend to reach higher performance. The learning rate was slowed as  $0.1(d / 50)^{-1 / 2}$  to account for observed instability in higher dimensions. The epoch was increased to  $5 \times (300 / |D|)^{1 / 2}$  for data size  $D$  in MB to run more epoch on smaller data size. We did not observe too much difference between this and default parameters beyond stabilizing higher dimensions.
vecmap-fasttext only fully works for the two cases with no domain mismatch – enwiki-dewiki and enparl-huparl. In contrast, coocmap-fasttext show that fasttext contains the information to tackle harder cases of enwiki-esnews, showing that while fasttext is retaining the necessary information, it does not have the necessary linear algebraic property for vecmap to work. On enparl-huparl, vecmap-fasttext did not work for 300 dimensions whereas coocmap-fasttext-clip did using the same vectors with clipping, achieving better accuracy than vecmap-fasttext 100 dim. On enwiki-esnews, only coocmap-fasttext-clip and coocmap-fasttext worked. Finally, all fasttext based methods failed on enwiki-hunews.
It is always possible that we are still using fasttext poorly, but we covered the main factors and clarified the role of dimension which seems sufficient to explain the observed behaviors.
C Clip, drop and truncate
Figure 7: Accuracy vs. dimension on ennews to zhwiki, on 100MB, 300MB and 1000MB of data.