SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sida I. Wang
FAIR,Meta
The striking ability of unsupervised word translation has been demonstrated recently with the help of low-dimensional word vectors / pretraining, which is used by all successful methods and assumed to be necessary. We test and challenge this assumption by developing a method that can also make use of high dimensional signal. Freed from the limits of low dimensions, we show that relying on low-dimensional vectors and their incidental properties miss out on better denoising methods and signals in high dimensions, thus stunting the potential of the data. Our results show that unsupervised translation can be achieved more easily and robustly than previously thought – less than 80MB and minutes of CPU time is required to achieve over  $50\%$  accuracy for English to Finnish, Hungarian, and Chinese translations when trained in the same domain; even under domain mismatch, the method still works fully unsupervised on English NewsCrawl to Chinese Wikipedia and English Europarl to Spanish Wikipedia, among others. These results challenge prevailing assumptions on the necessity and superiority of low-dimensional vectors and show that the higher dimension signal can be used rather than thrown away.
The ability to translate words from one language to another without any parallel data nor supervision has been demonstrated in recent works (Lample et al., 2018b; Artetxe et al., 2018b, ...), has long been attempted (Rapp, 1995; Ravi and Knight, 2011, ...), and provides empirical answers to scientific questions about language grounding (Bender and Koller, 2020; Søgaard, 2023). However, this striking phenomenon has only been demonstrated with the help of pretrained word vectors or transformer models recently, lending further support to the necessity and superiority of low-dimensional representations. In this work, we develop and test coocmap<sup>1</sup> for unsupervised word translation using only simple co-occurrence statistics easily computed from raw data. coocmap is dual method of vecmap (Artetxe et al., 2018b), using co-occurrences statistics in place of low-dimensional vectors. The greedy and deterministic coocmap establishes the most direct route from raw data to the striking phenomenon of unsupervised word translation, and shows that pretrained representation is not the key.
More surprisingly, coocmap greatly improves the data efficiency and robustness over the baseline of vecmap-fasttext, showing that relying on low-dimensional vectors is not only unnecessary but also inferior. With coocmap, 10-40MB of text data and a few minutes of CPU time is sufficient to achieve unsupervised word translation if the training corpora are in the same domain (e.g. both on Wikipedia, Figure 1). For context, this is less than the data used by Brown et al. (1993) to train IBM word alignment models. On cases that were reported not to work using unsupervised methods by Søgaard et al. (2018), we confirm their findings that using fasttext (Bojanowski et al., 2017) vectors indeed fails, while coocmap solved many of these cases with very little data as well. For our main results, we show that less similar language pairs such as English to Hungarian, Finnish and Chinese posed no difficulty and also start to work with 10MB of data for Hungarian, 30MB of data for Finnish
$^{1}$ code released at https://github.com/facebookresearch/coocmap
Figure 1: Our results focus on the data requirements as we varying the amount and domains of data. The unsupervised accuracy increases quickly with a sharp starting point point and this can happen surprisingly early ( $\sim$ 10MB here), reaching  $>50\%$  by 20MB. As data increases, supervised initialization with ground-truth dictionary (dict-init) does not make much difference as shown by the vanishing initialization gap. vecmap-fasttext needs more data and is less stable.
and 20MB of data for Chinese. The hardest case of training with domain mismatch (e.g. Wikipedia vs. NewsCrawl or Europarl) is generally reported not to work (Marchisio et al., 2020) especially for dissimilar languages. After aggressively discarding some information using clip and drop, coocmap works on English to Hungarian and Chinese, significantly outperforming vectors, which do not work.
These results challenge the conventional wisdom ever since word vectors proved their worth both for cross-lingual applications (Ruder et al., 2019) and more generally where a popular textbook claims dense vectors work better in every NLP task than sparse vectors (Jurafsky and Martin, 2023, 6.8). We discuss reasons that disadvantage low-dimensional vectors unless compression is the goal, which is far more intuitive than the conventional wisdom that dense vectors are both smaller and perform better. For vectors to perform, they must have nice linear-algebraic properties (Mikolov et al., 2013c,b), successfully denoise the data, while still retaining enough useful information. The linear algebraic and denoising properties are incidental, since they are not part of the training objective but are a consequence of being in low dimensions. These incidental properties come in conflict with the actual training objective to retain more information as dimension increases, leaving a small window of opportunity for success. The incidental denoising we get from having low dimensions, while interesting and sufficient in easy cases, is actually suboptimal and worse than the more intentional denoising in high dimensions used by coocmap. Furthermore, without the need to have low dimensions, coocmap can access useful information in higher dimensions. The higher dimensions contain knowledge such as portland-oregon, cbs-60 minutes, molecular-weight, luisiana-purchase, tokyo-1964 that tend to be lost in lower dimensions. We speculate that similarly processed co-occurrences would outperform low-dimensional vectors in other tasks too, especially if the natural but incidental robustness of vectors is not enough.
2 Problem formulation
This word translation task is also called lexicon or dictionary induction in the literature. The task is to translate words in one language to another (e.g. hello to bonjour in French) and is evaluated for accuracy (precision@1) on translations of specific words in a reference dictionary. We do this fully unsupervised, meaning we do not use any seed dictionary or character information. Let the datasets  $D_{1}$  and  $D_{2}$  be sequences of words from the vocabulary sets  $V_{1}, V_{2}$ . Since we do not use character information, we may refer to vocabulary by their integer indices  $V = [1, 2, \ldots]$  for convenience. We would like to find mapping from  $V_{1}$  to  $V_{2}$  based on statistics of the datasets. In particular, we consider the window model where the sequential data is reduced to pairwise co-occurrences over context windows of a fixed size  $m$ . The word-context co-occurrence matrix  $\mathrm{Co} \in \mathbb{R}^{|V| \times |V|}$  counts the number of times word  $w$  occurs in the context of  $c$ , over a window of some size  $m$
$$
\operatorname {C o} (w, c) = \sum_ {i = 1} ^ {| D |} \sum_ {- m \leq j \leq m, j \neq 0} \mathbb {I} [ w _ {i} = w, w _ {i + j} = c ] \tag {1}
$$
where  $w_{i}$  is the  $i$ -th word of the dataset  $D$ . The matrix  $\mathrm{Co}$  is the sufficient statistics of the popular word2vec (Mikolov et al., 2013a,c) and fasttext (Bojanowski et al., 2017), which use the same
co-occurrence information, including additional objectives not explicit in the loss function:
$$
\ell (\theta) = \sum_ {i = 1} ^ {| D |} \sum_ {- m \leq j \leq m, j \neq 0} \log_ {\theta} p \left(w _ {i + j} \mid w _ {i}\right). \tag {2}
$$
For word translation, we obtain  $\mathrm{Co}_1$  and  $\mathrm{Co}_2$  from the two languages  $D_{1}$  and  $D_{2}$  separately.
Multilingual distributional hypothesis. If words are characterized by its co-occurrences (Harris, 1954), then translated words will keep their translated co-occurrences. In more precise notation, let  $(s_1, t_1), (s_2, t_2), \ldots, (s_n, t_n)$  be  $n$  pairs of translated words, then for translation  $(s, t)$
$$
X [ s, s _ {1} ], X [ s, s _ {2} ], \ldots , X [ s, s _ {n} ] \sim Z [ t, t _ {1} ], Z [ t, t _ {2} ], \ldots , Z [ t, t _ {n} ],
$$
for association matrices  $X = K(\mathrm{C}_{0_1})$ ,  $Z = K(\mathrm{C}_{0_2})$ . While intuitive, this is not known to work unsupervised nor with minimal supervision. Rapp (1995) presents evidence and correctly speculates that there may be sufficient signal. Fung (1997); Rapp (1999) used a initial dictionary to successfully expand the vocabulary further using a mutual information based association matrix. Alvarez-Melis and Jaakkola (2018) operates on an association matrix generated from word vectors.
Isomorphism of word vectors. Despite the clear motivation in the association space, unsupervised translation was first shown to work in vector space, where a linear mapping between the vector spaces corresponds to word translation. This is called (approximate) isomorphism (Ruder et al., 2019). If  $s$  translate to  $t$ , and  $X_s, Z_t$  are their vectors, then  $X_sW \sim Z_t$  where  $W$  can be a rotation matrix, and the association metric can be cosine-distance, often after applying normalizations to raw word vectors. Successful methods solve this problem using adversarial learning (Lample et al., 2018b) or heuristic initialization and iterative refinement (Artetxe et al., 2018b; Hoshen and Wolf, 2018), among others (Zhang et al., 2017b).
We aim for simplicity in coocmap, and all steps consist of discrete optimization by arranging columns of  $X$  or measuring distances between rows of  $X$  where  $X$  is the association matrix based on co-occurrences (and  $Z$  for the other language). Almost all operations of coocmap has an analog in vecmap and we will describe both of them for clarity. There are two main steps in both methods, finding best matches and measuring cdist pairwise distances. Each row of  $X$  corresponds to a word for both vecmap and coocmap and are inputs to cdist. vecmap finds a rotation  $W$  that best match given row vectors  $X[s,:]$  and  $Z[t,:]$  for  $s = [s_1, s_2, \ldots, s_n] \in V_1^n$  and  $t = [t_1, t_2, \ldots, t_n] \in V_2^n$ . For coocmap, instead of solving for  $W$ , we just re-arrange the columns of the association matrix with indices  $s, t$  directly to get  $X[:s], Z[:t]$  where the columns of  $X$  are words too.
Algorithm 1 coocmap self-learning
[ X \in \mathbb{R}^{|V_1| \times |V_1|}, Z \in \mathbb{R}^{|V_2| \times |V_2|}; ]
Input  $s,t$  Output  $s,t$
$$
D = \operatorname {c d i s t} (X [:, s ], Z [:, t ])
$$
$$
s, t = \operatorname {m a t c h} (D)
$$
until no more improvement
Algorithm 2 vecmap self-learning
$X\in \mathbb{R}^{|V_1|\times d},Z\in \mathbb{R}^{|V_2|\times d};$
$$
W = \operatorname {s o l v e} (X [ s,: ], Z [ t,: ])
$$
$$
D = \operatorname {c d i s t} (X W, Z)
$$
$\operatorname{cdist}(X,Z)_{ij} = \operatorname{dist}(X_i,Z_j)$  is a function that takes pairwise distances for each row of both inputs and then outputs the results in a matrix. The self-learning loop uses the same improvement measurement as in vecmap mean $_i$ $\max_j$  ( $\operatorname{cdist}(i,j)$ ), but without stochastic search. We explain how to generate  $X, Z, s, t$ , and match next.
Measurement. It is important to normalize before taking measurements, and vecmap normalization consists of normalizing each row to have unit  $\ell_2$ -norm (unitr), center so each column has 0 mean (centerc), and normalizing rows (unitr) again. For precision,
$$
\operatorname {n o r m a l i z e} (Y) := \operatorname {u n i t r} (\operatorname {c e n t e r c} (\operatorname {u n i t r} (Y))), \tag {3}
$$
where  $\operatorname{centerc}(Y) := Y - \operatorname{sumr}(Y) / r$ ,  $\operatorname{sumr}(Y) := \mathbf{1}^T Y$ , and  $Y \in \mathbb{R}^{r \times c}$ .
2We show a derivation in E on the equivalence of vecmap and coocmap.
**coocmap.** The input matrix  $X = \text{normalize}(\operatorname{Co}^{\circ \frac{1}{2}})$  is obtained from Co by taking elementwise square-root then normalize. cdist is the cosine-distance.
vecmap. From original word vectors  $X'$ , we use their normalized versions  $X = \mathrm{normalize}(X')$ . For solve  $(X[s], Z[t])$ , we use  $\arg \min_{W \in \Omega} ||X[s, :]W - Z[t, :]||_F$ , for rotation matrices  $\Omega$ . This is known as the Procrustes problem. cdist is the cosine-distance.
Initialization. We need the initial input  $s, t$  for Algorithms 1 and 2. For the unsupervised initialization, we follow vecmap's method based on row-wise sorting. Let sort_row(X) make each row of  $X$  be sorted independently and normalize(X) is defined in (3), then
Algorithm 3 unsupervised initialization
Input  $X,Z$  Output  $D$
$$
R (Y) := \operatorname {s o r t} _ {\text {r o w}} (K (Y))
$$
$$
S (Y) := \operatorname {n o r m a l i z e} (R (Y))
$$
$$
D = \operatorname {c d i s t} (S (X), S (Z))
$$
vecmap:  $X,Z\in \mathbb{R}^{|V|\times |V|}$
$$
X = \left(X ^ {\prime} X ^ {\prime \intercal}\right) ^ {\frac {1}{2}} \text {f o r} X ^ {\prime} \in \mathbb {R} ^ {| V | \times d}
$$
coocmap:  $X, Z \in \mathbb{R}^{|V| \times |V|}$
$$
X = \mathrm {n o r m a l i z e} (\mathrm {C o} _ {1} ^ {\circ 1 / 2})
$$
The first step of vecmap  $X = (Y'Y'^{\intercal})^{\frac{1}{2}}$  is actually converting from vector space to the association space. So it is natural to replace the first step by normalize  $(\mathrm{Co}_1^{\circ 1/2})$  for coocmap.
Matching. The main problem once we have a distance matrix is to solve a matching problem
$$
\min  _ {M} \sum_ {(i, j) \in M} \operatorname {c d i s t} (i, j) \tag {4}
$$
vecmap proposes a simple matching method, where we take  $j^{*} = \arg \min_{j}\operatorname {cdist}(i,j)$  for each  $i$  in forward matching, and then take  $i^{*} = \arg \min_{i}\operatorname {cdist}(i,j)$  for each  $j$  in backward matching. This always results in  $|V_1| + |V_2|$  matches where words on each side is guaranteed to appear at least once. For coocmap, there is complete freedom in forming matches  $i,j$  and often many words all match with a single word. As a result, hubness mitigation (Lazaridou et al., 2015) is even more essential compared to vecmap.
While there are many reasonable matching algorithms, we find that Cross-Domain Similarity Local Scaling (CSLS) (Lample et al., 2018b) was enough to stabilize coocmap. Note that CSLS acts on the pairwise distances and therefore directly applies to cdist,
$$
\operatorname {c s l s} (\operatorname {c d i s t} (i, j)) = \operatorname {c d i s t} (i, j) - \frac {1}{2 k} \left(\sum_ {j ^ {\prime} \in N _ {i} (k)} \operatorname {c d i s t} (i, j ^ {\prime}) + \sum_ {i ^ {\prime} \in N _ {j} (k)} \operatorname {c d i s t} (i ^ {\prime}, j)\right)
$$
where  $N_{i}(k)$  are the  $k$ -nearest neighbors to  $i$  according to cdist. We use csls(cdist) as the input to (4) in place of cdist. Instead of always preferring the best absolute match, CSLS prefers matches that stand out relative to the  $k$  best alternatives.
3.1 Clip and drop
This basic coocmap above already works well on the easier cases and these techniques provided small improvements. However, for difficult cases of training on different domains, clip is essential and drop gives a final boost that can be large. These operations are aggressive and throws away potentially useful information just like rank reduction, but we will show that they are far better especially under domain shifts.
clip. For clip, we threshold the top and bottom values by percentile. While there are just 2 numbers for the 2 thresholds, we use two percentiles operations to determine each of them so they are not dominated by any particular row. Let  $r_i = Q_p(X_i)$  be the  $p$ th percentile value of the row  $i$  of  $X$ . We threshold  $X$  at  $Q_p(r_1, r_2, \ldots, r_{|V|})$ , where  $p = 1\%$  for lowerbound and  $p = 99\%$  for upperbound. This results in two thresholds for the entire matrix that seem to agree well across languages. For results on clip, we run coocmap with  $X = \mathrm{clip}(\mathrm{normalize}(\mathrm{Co}^{\circ 1/2}))$ . Intuitively, if words are already very strongly associated, obscressing over exactly how strongly is not robust across languages and datasets but can incur large  $\ell_2$ s. For lowerbound, extremely negatively associated words is clearly not robust, since the smallest count is 0 and one can always use any two words together. Both bounds seem to improve over baseline, but the upperbound is the more important one.
drop (head). Drop the  $r = 20$  largest singular vectors, i.e.  $\mathrm{drop}_r(X) = X - X_r\in \mathbb{R}^{|V|\times |V|}$ , where  $X_{r}$  is the rank-  $r$  approximation of  $X$  by SVD. In the experiments, we first get the solution of  $s,t$  from  $\mathrm{clip}(\mathrm{normalize}(\mathrm{Co}^{\circ 1 / 2}))$  then use coocmap on  $X = \mathrm{clip}(\mathrm{drop}_r(\mathrm{normalize}(\mathrm{Co}^{\circ 1 / 2})))$  with  $s,t$  as the initial input. Drop was good at getting a final boost from a good solution, but is usually worse than the basic coocmap in obtaining an initial solution. While the dropping top 1 singular vector is discussed by (Mu and Viswanath, 2018; Arora et al., 2017) and popular vectors implicitly drop already (see A), we see more benefits from dropping more and this seems to enable more benefits of the higher dimensions. We show examples in Appendix F in support of this viewpoint.
Truncate (tail). This is the usual rank reduction where  $\mathrm{trunc}_r(X) = X_r$  is the best rank- $r$  approximation of  $X$  by SVD. We use this for analysis on the effect of rank.
For the main results, we report accuracy as a function of data size and only show results in the fully unsupervised setting. The accuracy is measured by precision at 1 on the full MUSE dictionary for the given language pair on the 5000 most common words in each language (see Section 6 for limitations). Many results will be shown as scatter plots of accuracy vs. data size, each containing thousands of experiments and more informative than tables. They will capture stability and the qualitative behaviors of the transitional region as the amount of data varies. Each point in the scatter plots represents an experiment where a specific amount of data was taken from the head of the file for training co-occurrence matrices and fasttext vectors with default settings (skipgram, 300 dimension, more in B) for fasttext. coocmap use the same window size as fasttext ( $m = 5$ ), the same CSLS ( $k = 10$ ) and same optimization parameters as vecmap. coocmap does not require additional hyperparameters until we add clip and drop. The same amount of data is taken from both sides unless one side is exhausted. In our experiments, most cases either achieve  $>50\%$  accuracy (i.e. works) or near 0 accuracy and has a definite starting point (i.e. starts to work). Summary of these results are in Table 1 and we will show details on progressively harder tests.
Methods. we compare these methods for the main results.
- dict-init: initialize with the ground truth dictionary then apply coocmap.
- coocmap: fully unsupervised coocmap and improvements with -clip, -drop if needed.
- vecmap-fasttext: apply vecmap to 300 dimensional fasttext vectors trained with default settings.
- vecmap-raw: apply vecmap to a svd factorization of the co-occurrence matrix. If  $\mathrm{Co}^{\circ 1/2} = USV'$ , then we use  $US_{r}$  as the word vectors where  $S_{r}$  only keeps top  $r$  singular values.  $r = 300$ .
For dict-init, we initialized using the ground truth dictionary (i.e. initial input  $s$ ,  $t$  to Algorithm 1 are the true evaluation dictionary) and then test on the same dictionary after self-learning. This establishes an upper-bound for the amount of gains possible with a better initialization as long as we are using the basic coocmap measurements and self-learning. After coocmap works stably, their performance coincides, showing very few search failures and the limit of gains from initialization.
We use default parameters for fasttext in this section. This may be unfair to fasttext but better match other results reported in the literature where the default hyperparameters are used or pretrained vectors are downloaded. In Section B, we push on the potential of fasttext more and describe hyperparameters.
Data. We test on these languages paired with English (en): Spanish (es), French (fr), German (de), Hungarian (hu), Finnish (fi) and Chinese (zh).
For training data we use Wikipedia (wiki), Europarl (parl), and NewsCrawl (news), processed from the source and removing obvious markings so raw text remains. The data is processed using Huggingface WordLevel tokenizer with whitespace pre-tokenizer and lower-cased first.
wiki (https://dumps.wikimedia.org/): Wikipedia downloaded directly from the official dumps (pages-meta-current), extract text using WikiExtractor (Attardi, 2015) and removed <doc id tags. We start from the first dump until is  $>1000\mathrm{MB}$  of data for each language and shuffle the combined file. For zh, we also strip away all Latin characters [a-zA-Z] and segment using jieba: https://github.com/fxsjy/jieba.
parl (https://www.statmt.org/european/): Europarl (Koehn, 2005) was shuffled after downloading since this is parallel data.
news (https://data.statmt.org/news-crawl/): NewsCrawl 2019.es was downloaded and used as is. For 2018-2022.hu and 2018.en, we striped meta data by grep -v on http, trackingCode and { after which a small random sample does not obviously contain more meta data. This removed over  $35\%$  from hu news and  $0.1\%$  from en.
For evaluation data, we use the full MUSE dictionaries for these language pairs. This allows for a more stable evaluation given our focus on qualitative behaviors, but at a cost of being less comparable to previous results.
source	target	start	start'	works
enwiki	eswiki	9	70	20
enwiki	frwiki	10	80	30
enwiki	zhwiki	14	700	50
enwiki	dewiki	20	200	70
enparl	huparl	8	-	20
enparl	fiparl	30	-	80
Domain mismatch
source	target	start	start'	works
enwiki	esnews	30	-	70
enwiki	hunews	140	-	600
enwiki	esparl	500	-	500
ennews	zhwiki	800	-	800
enwiki	huparl	-	-	-
enwiki	fiparl	-	-	-
Table 1: Summary of data requirements for coocmap in MB (1e6 bytes). start: when coocmap starts to work, start': when vecmapfasttext baseline starts to work (check Figure 2 to see that start is clear), - denotes failure for the whole range; works: when coocmap reaches  $50\%$  accuracy. Readings are rounded up to the next tick of the log plot, or 1.4 if less than the middle of 1 and 2. The same amount of data is used on both source and target sides, unless one side is used up (e.g. 100MB of huparl or 300MB of esparl). There are less than 0.2 million tokens per MB in all datasets, ranging from 0.13 in fiparl, 0.19 in zhwiki and 0.20 for ennews.
4.1 Same domain of data
Figure 2: accuracy vs. datasize where the source data and target data are in the same domain, either both Wikipedia, or both Europarl (enparl-huparl, and enparl-fiparl). These cases are easy for coocmap but sometimes failed for vecmap-fasttext or required more data.
Similar languages. To establish some baselines, we start with the easy settings where everything eventually works. Figure 2 shows that coocmap starts to work at around 10MB of data while vecmap trained on fasttext only starts to work once there are over 100MB of data. The transitional region is fairly sharp, and coocmap works reliably on each language pair with around 100MB of data, and vecmap with fastText also mostly works once there is around 100MB of data. vecmap-fasttext eventually outperforms coocmap after it was trained on  $>100\mathrm{MB}$  of data.
Less similar languages. For hard cases from (Søgaard et al., 2018), all of which were reported not to work using MUSE (Lample et al., 2018b) using fastText vectors. Here, we confirm that vecmap and fastText vectors also do not work on default settings. However, these are not even challenging to the basic coocmap where all cases started to work with less than 30MB of data.