Accessing Higher Dimensions for Unsupervised Word Translation
Authors: Sida Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: co-occurrences
unsupervised word translation
bilingual lexicon induction
robust statistics
unsupervised machine translation
TL;DR: use the information contained in higher dimension that is usually tossed by word presentations to get more robust and data efficient unsupervised word translation

Abstract: The striking ability of unsupervised word translation has been demonstrated recently with the help of low-dimensional word vectors / pretraining, which is used by all successful methods and assumed to be necessary. We test and challenge this assumption by developing a method that can also make use of high dimensional signal. Freed from the limits of low dimensions, we show that relying on low-dimensional vectors and their incidental properties miss out on better denoising methods and signals in high dimensions, thus stunting the potential of the data. Our results show that unsupervised translation can be achieved more easily and robustly than previously thought -- less than 80MB and minutes of CPU time is required to achieve over 50\% accuracy for English to Finnish, Hungarian, and Chinese translations when trained in the same domain; even under domain mismatch, the method still works fully unsupervised on English NewsCrawl to Chinese Wikipedia and English Europarl to Spanish Wikipedia, among others. These results challenge prevailing assumptions on the necessity and superiority of low-dimensional vectors and show that the higher dimension signal can be used rather than thrown away.

Accessing Higher Dimensions for Unsupervised Word Translation
Sida I. Wang
FAIR, Meta
Abstract
The striking ability of unsupervised word translation has been demonstrated re- cently with the help of low-dimensional word vectors / pretraining, which is used by all successful methods and assumed to be necessary. We test and challenge this assumption by developing a method that can also make of high dimensional signal. Freed from the limits of low dimensions, we show that relying on low- dimensional vectors and their incidental properties miss out on better denoising methods and signals in high dimensions, thus stunting the potential of the data. Our results show that unsupervised translation can be achieved more easily and robustly than previously thought - less than 80MB and minutes of CPU time is required achieve over 50% accuracy for English Finnish, Hungarian, and Chinese translations when trained in the same domain; even under domain mismatch, the method still works fully unsupervised and English Europarl to Spanish Wikipedia, among others. These results challenge prevailing assumptions on the necessity and superiority of low-dimensional vectors and show that the higher dimension signal can be used rather than thrown away.
English NewsCrawl to Chinese Wikipedia
Introduction
The ability to translate words from one language to another without any parallel data nor supervision has been demonstrated in recent works (Lample al., 2018b; Artetxe et al., 2018b, ...), has long been attempted (Rapp, 1995; Ravi and Knight, 2011.... ). and provides empirical answers to scientific questions about language grounding (Bender and Koller, 2020; Søgaard, 2023). However, this striking phenomenon has only been demonstrated with the help of pretrained word vectors or transformer models recently, lending further support to the necessity and superiority of low- dimensional representations. In this work, we develop and test coocmap for unsupervised word translation using only simple co-occurrence statisties easily computed from raw data. coocmap is dual method of vecmap (Artetxe et al., 2018b), using co-occurrences statistics in place of low-dimensional vectors. The greedy and deterministic coocmap establishes the most direct route from raw data to the striking phenomenon of unsupervised word translation, and shows that pretrained representation is not the key.
More surprisingly, cooemap greatly improves the data efficiency and robustness over the baseline of fasmtp-fasttext, showing that relying on low-dimensional vectors is not only unnecessary but also inferior. With coocmap, 10-40MB of text data and few minutes of CPU time is sufficient to achieve unsupervised word translation if the training corpora are in the same domain (e.g. both on Wikipedia, Figure 1). For context, this is less than the data used by Brown et al. (1993) train IBM word alignment models. On cases that were reported not to work using unsupervised methods by Søgaard et al. (2018), we confirm their findings that using fasttext (Bojanowski et al., 2017) vectors indeed fails, while coocmap solved many of these cases with very little data as well. For our main results, we show that less similar language pairs such as English to Hungarian, Finnish and Chinese posed no difficulty and also start to work with 10MB of data for Hungarian, 30MB of data for Finnish
I code relcased at https: http::///gttaaa..c///aaaaaaaoaap http:///....ccccaooeeearaaaaraaa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Swwiki-eswik
coocmap
Figure Our results focus on the data re- quirements as we varying the amount and do- mains of data. The unsupervised accuracy increases quickly with sharp starting point point and this can happen surprisingly early (~10MB here), reaching >50% by 20MB, As data increases, supervised initialization with ground-truth dictionary (dict-init) does not make much difference as shown by the vanish- ing initialization gap. vecmap-fastter needs more data and is less stable.
init
50% occ. at 20MB
0.4
init. gop
0.2
storts to work at 9MB
100 10
307
162
and 20MB of data for Chinese. The hardest case of training with domain mismatch (e.g. Wikipedia vs. NewsCrawl or Europarl) is generally reported not to work (Marchisio et al., 2020) especially for dissimilar languages. After aggressively discarding some information using clip and drop, cooemap works on English to Hungarian and Chinese, significantly outperforming vectors, which do not work.
These results challenge the conventional wisdom ever since word vectors proved their worth both for cross-lingual applications (Ruder et al., 2019) and more generally where a popular textbook claims dense vectors work better in every NLP task than sparse vectors (Jurafsky and Martin, 2023, 6.8). We discuss reasons that disadvantage low-dimensional vectors unless compression is the goal, which is far more intuitive than the conventional wisdom that dense vectors are both smaller and perform better. For vectors to perform, they must have nice linear-algetee aleebbac properties (Mikolov et al., 2013c,b), successfully denoise the data, while still retaining enough useful information. The linear algebraic and denoising properties are incidental, since they are not part of the training objective but are consequence of being in low dimensions. These incidental properties come in conflict with the actual training objective to retain more information as dimension increases, leaving small window of opportunity for success. The incidental denoising get from having low dimensions, while interesting and sufficient in easy cases, is actually suboptimal and worse than the more intentional denoising in high dimensions used by coocmap. Furthermore, without the need to have low dimensions, coocmap can access useful information in higher dimensions. The higher dimensions contain knowledge such as portland-ores ooegon,, cbs-60 minutes, molecular-weight luisiana- tuisiana-purchase, tokyo-1964 that tend to be lost in lower dimensions. We speculate that similarly processed co-occurrences would outperform low-dimensional vectors in other tasks too, especially the natural but incidental robustness of vectors is not enough.
Problem formulation
This word translation task is also called lexicon or dictionary induction in the literature. The task is to translate words in one language another (e.g. hello bonjour in French) and is evaluated for accuracy (precision@ @ I) on translations of specific words in a reference dictionary. We do this fully unsupervised, meaning we do not any seed dictionary character information. Let the datasets D, and D2 be sequences of words from the vocabulary sets Vi, V/2 Since we do not use character information, we may refer vocabulary by their integer indices V = [1, 2, ...] for convenience. We would like to find mapping from Vi to V/2 based on statistics of the datasets. In particular, we consider the window model where the sequential data is reduced to pairwise co-occurrences over context windows of fixed size m. The word-context co-occurrence matrix Co RVVx[VV counts the number of times word w occurs in the context of c, over a window of some size m
D
Co(w, c)
Σ i=1
Σ
Iww w. WH+j cj
where un the i-th word of the dataset The matrix Co the sufficient statistics of the popular word2vee (Mikolov et al., 30,c) and fasttext (Bojanowski et al., 2017), which use the same

---

co-occurrence information, including additional objectives not explicit in the loss function:
((0)= = Σ Σ
loge p p(ur+)//u.)
==1 -SSmmJ7
For word translation, we obtain Co and Coz from the two languages D1 and D2 separately.
Multilingual distributional hypothesis. If words are characterized by its co-occurrents (Harris, 1954), then translated words will keep their translated co-occurrents. In more precise notation, let (51,1),, (82, f2),.... (8n; (n) be pairs of translated words, then for translation (s,
X[s, s1 82], X|s, sn] ~ ztt
2tt,11) (1), Ztt, /2],-..: Z(t, 1,)]
for association matrices X = (Coi)), Z = (Coo)). While intuitive, this is not known work unsupervised nor with minimal supervision. Rapp (1995) presents evidence and correctly speculates that there may be sufficient signal. Fung (1997); Rapp (1999) used initial dictionary to successfully expand the vocabulary further using a mutual information based association matrix. Alvarez-Melis association matrix generated from word vectors.
and Jaakkola (2018) operates
Isomorphism word vectors. Despite the clear motivation the association space, unsupervised translation was first shown to work in vector space, where linear mapping between the vector spaces corresponds to word translation. This is called (approximate) isomorphism (Ruder et al., 2019). translate to f, and X,, Zi are their vectors, then X W Z₁ where W can be rotation matrix, and the association metric can be distand, often after applying normalizations to raw word vectors. Successful methods solve this problem using adversarial learning (Lample et al., 2018b) or heuristic initialization and iterative refinement (Artetxe et al., 2018b; Hoshen and Wolf, 2018), among others (Zhang et al., 2017b).
3 Method
We aim for simplicity in cooemap, and all steps consists of discrete optimization by arranging columns of X or measuring distances between rows of X where X the association matrix based on co-occurrences (and Z for the other language). Almost all operations of coocmap has an analog in veemap and we will describe both of them for clarity. There are two main steps in both methods, finding best matches and measuring edist pairwise distances. Each row of X corresponds to word for both vecmap and coocmap and are inputs to odist. vecmap finds rotation W that best match given row vectors X [[s.:] and Zit, :] for For coocmap, instead solving for W, we just re-arrange the columns of the association matrix with indices directly to get X(: s), Z where the columns of X are words too.?
[$1, 82,..., E V/" and
t2,12... tt.) E Vi
Algorithm 1 coocmap self- self-learning
Algorithm 2 vecmap self- self-learning
X E IR [i Z E R|/]|xIVll; Input s,t Output s,t repeat
X E R Vi|xd Z E R [V/xxd
Input s,t Output s,t repeat
W sol ve( X [18 : Z (1:
D odist(X" s). Z[: ()
D cdist (XW. 7)
8, / match( D)) until more improvement
8, 1 = match (D)
until more improvement
cdist(X, 2); dis Zj) is a function that takes pairwise distances for each row of both inputs and then outputs the results in matrix. The self-learning loop the same improvement measurement as in vecmap mean, max; maxy (cdist()(). but without stochastic search. We explain how to generate X, Z, 8, and match next.
Measurement. It is important to normalize before taking measurements, and vecmap normalization consists of normalizing each row to have unit fo-m (unitr), center so each column has mean (centerc), and normalizing rows (unitr) again. For precision,
normalize(Y)) == unitr (nitt((ceterc((uni((()) where centerc(Y) (Y) == (is-uumnYY//s sumr(Y) 1"YY, and Y E grxe
We show derivation in E on the equivalence of vecmap and coocmap.
3

---

coocmap. The input matrix X normalize(Co" is obtained from Co by taking elementwise square-root then normalize. cdist the cosine-distance.
vecmap. From original word vectors X', we use their normalized versions normalize( X For (X)) f())), we use arg minwes en [x :lw =[l,: for rotation matrices f2 This is known as the Procrustes problem. cdist the cosine distanedsstance.
Initialization. We need the initial imput s, for Algorithms 1 and 2. For the unsupervised initializa- tion, we follow vecmap's method based on row-wise sorting. Let sort row make each row X be sorted independently and normalize( X) is defined in (3), then
vecmap: X, E TRRVV
Algorithm 3 unsupervised initialization
X (X' for X' E BRV
Input X, Z, Output D
R(Y) == sort_ row (Y)))
cooemap: X, ERIVI X = Normalize(Co)" (Co, 1//)
S(Y)= normalize R(Y))
D = cdist(S(X), S(Z)
The first step of vecmap X (Y'Y'))& actually converting from vector space to the association
space. So is natural to replace the first step by normalize Co2 21/2) for cooemap.
Matching. The main problem once we have a distance matrix is solve a matching problem
min M
Σ
ci,
(.J)EM
vecmap proposes simple matching method, where take j arg miny edist(i,)) j) for each in forward matching, and then take i* = arg min, edist(i,j) for each backward matching. This always results in [V] |V2| matches where words each side is guaranteed to appear least once. For cooemap, there is complete freedom in forming matches i,j and often many words all match with single word. As result, hubness mitigation (Lazaridou al., 2015) is even more essential compared to vecmap.
While there are many reasonable matching algorithms, we find that Cross-Domain Similarity Local Scaling (CSLS) (Lample et al., (0) was enough to stabilize cooemap. Note that CSLS acts on the pairwise distances and therefore directly applies to edist,
csls(cdist((,.)) = cdi 2k
Σ J'EN.(K)
Σ cdist ()) (EN,(())
where (k) are the k-nearest neighbors i according edist. We use csls(cdist)) the input to (4) in place of cdist. Instead of always preferring the best absolute match, CSLS prefers matches that stand out relative the & best alternatives.
3.1 Clip and drop
This basic coocmap above already works well on the easier cases and these techniques provided small improvements. However, for diffiuutt cases of training on different domains, clip is essential and drop gives final boost that can be large. These operations are aggressive and throws away potentially useful information just like rank reduction, but we will show that they are far better especially under domain shifts.
clip. For clip, we threshold the top and bottom values by percentile. While there are just 2 numbers for the 2 thresholds, we use percentiles operations determine each of them they dominated by any particular row. Let = Qp be the pth percentile value of the row i X. We threshold X (2.......... .........j where p = 1% for lowerbound and p = 99% for upperbound. This results in two thresholds for the entire matrix that seem to agree well across languages, For results on clip, we run coocmap with X clip(normalize( 11/2). Intuitively. if words are already very strongly associated, obsessing over exactly how strongly is not robust across languages and datasets but can incur large lss. For lowerbound, extremely negatively associated words is clearly not robust, since the smallest count 0 and one can always use any two words together. Both bounds seem to improve over baseline, but the upperbound is the more important one.

---

drop (head). Drop the 20 largest singular vectors, i.e. (x), X - X, E RiVII, where X, is the rank-r approximation of X by SVD. In the experiments, we first get the solution of t from clip (normalizz((o" then use cooemap on X = clip(drop, Coormalize(Coo!// with s, É as the initial input. Drop was good at getting a final boost from a good solution, but usually worse than the basic coocmap in obtaining an initial solution. While the dropping top 1 singular vector is discussed by (Mu and Viswanath, 2018; Arora al., 2017) and popular vectors implicitly drop already (see A), we see more benefits from dropping more and this seems to enable more benefits of the higher dimensions. We show examples in Appendix in support of this viewpoint.
Truncate (tail). This is the usual rank reduction where trunc, approximation of X by SVD. We use this for analysis on the effect of rank.
X, is the best rank-r
4 Experiments
For the main results, we report accuracy as function of data size and only show results in the fully unsupervised setting. The accuracy is measured by precision at 1 on the full MUSE dictionary for the given language pair on the 5000 most common words in each language (see Section 6 for limitations). Many results will be shown as scatter plots of accuracy vs. data size, each containing thousands of experiments and more informative than tables. They will capture stability and the qualitative behaviors of the transitional region as the amount of data varies, Each point in the scatter plots represents an experiment where a specific amount of data was taken from the head of the file for training co-occurrence matrices and fasttext vectors with default settings (skipgram, 300 dimension, more in B) for fasttext. cooemap use the same window size as fasttext (m = 5), the same CSLS (k = 10) and same optimization parameters vecmap. coocmap does not require additional hyperparameters until we add clip and drop. The same amount of data is taken from both sides unless one side is exhausted. In our experiments, most cases either achieve 50% accuracy (i.e. works) or near accuracy and has a definite starting point (i.c. starts to work). Summary of these results are in Table I and we will show details on progressively harder tests.
Methods. we compare these methods for the main results.
dict-init: initialize with the ground truth dictionary then apply coocmap.
coocmap: fully unsupervised cooemap and improvements with -clip. -drop if needed.
veemap-fasttext apply vecmap to 300 dimensional fasttext vectors trained with default settings.
vecmap-raw: apply vecmap to a svd factorization of the co-occurence matrix. Co = USV', then we use U.S as the word vectors where S, only keeps top r singular values.
300.
For dict-init, we initialized using the ground truth dictionary (i.e. initial input 8, to Algorithm the true evaluation dictionary) and then test the same dictionary after self-lcarning. This establishes an upper-bound for the amount of gains possible with a better initialization as long as we are using the basic coocmap measurements and self-learning After coocmap works stably, their performance coincides, showing very few search failures and the limit of gains from initialization.
We use default parameters for fasttext in this section. This may be unfair fasttext but better match other results reported in the literature where the default hyperparameters are used or pretrained vectors are downloaded. In Section B. we push the potential of fasttext more and describe hyperparameters.
We test on these languages paired with English (en): Spanish (es), French (fr), German (de), Hungarian (hu), Finnish (fi) and Chinese (zh).
Data.
For training data we use Wikipedia (wiki), Europarl (parl), and NewsCrawl (news), processed from the source and removing obvious markups so raw text remains. The data is processed using Huggingface Wordl.eve tokenizer with whitespace pre-tokenizer and lower-cased first.
wiki (httpe: //dumps wikimedia org/): Wikipedia downloaded directly from the official dumps ppages-meta-curren)) extract text using WikiExttractor (Attardi, 2015) and removed <doc id tags. We start from the first dump until is > 1000MB of data for each language and shuffle the com- bined file. For zh, we also strip away all Latin characters [a-zA-Z] and segment using jieba: https://gitub..com//iiiiiiiii
parl thttps://war statmt........ waatttooog/egrr////.// Europarl (Koehn, 2005) was shuffled after download- ing since this is parallel data.
5

---

news (https://data. statmt. wwwp:/us.oo/eeoeeeeeeeeoo crawl/): NewsCrawl 2019.es was downloaded and used as is. For 2018-2022.hh hu and 2018.en, we striped meta data by grep -v on http, trackingCode and { after which small random sample does not obviously contain more meta data. This removed over 35% from hu news and 0.1 %% from en.
For evaluation data, we use full MUSE dictionaries for these language pairs. This allows for more stable evaluation given our focus on qualitative behaviors, but at a cost of being less comparable to previous results.
source enwiki
target
start
start'
works
Table I: Summary of data requirements for coocmap in MB (1e6 bytes). start: when coocmap starts work, start': when vecmap- fasttext baseline starts to work (check Fig- ure 2 to see that start clear), denotes failure for the whole range; works: when cooemap reaches 50% accuracy. Readings are rounded up to the next tick of the log plot, or 1.4 if less than the middle of I and 2. The same amount of data is used on both source and target sides, unless one side is used up (e.g. 100MB of huparl or 300MB of esparl). There are less than 0.2 million tokens per MB in all datasets, ranging from 0.13 in fi- parl, 0. in zhwiki and 0.20 for ennews.
eswiki
9
70
20
enwiki
frwiki
10
80
30
enwiki
zhwiki
14
700
50
enwiki
dewiki
20
200
70
enparl
huparl
8
20
enparl
fiparl
30
80
Domain mismatch
source
target
start
start'
works
enwiki
esnews
30
70
enwiki
hunews
140
600
enwiki
esparl
500
500
ennews
zhwiki
800
800
enwiki
huparl
enwiki
fiparl
4.1
Same domain of data
srink voww
enwic frw kj
clnck koW k)
ciccina vecrop-tationn
clovie
31-
ciciia
fertier
Fertee
sacruprin
3.4
12:
esw kidhii
erparitupar
eppor-fppal
677
62
61
162
Figure 2: accuracy vs. datasize where the source data and target data are in the same domain, either both Wikipedia, or both Europarl (enparl-huparl, and enparl-fiparl.. These cases easy for cooemap but sometimes failed for recmap-fasttex: or required more data.
Similar languages. To establish some basclines, we start with the easy settings where everything eventually works. Figure 2 shows that cooemap starts to work at around 10MB of data while vecmap trained on fasttext only starts to work once there are over 100MB of data. The transitional region is fairly sharp, and coocmap works reliably on each language pair with around 100MB of data, and vecmap with fastTexx also mostly works once there around 100MB of data. veemap-fasttext eventually outperforms coocmap after it was trained on > >10MMB of data.
Less similar languages. For hard cases from (Søgaard et al., 2018), all of which were reported not to work using MUSE (Lample et al., 201 1b)8)) using fastText vectors. Here, we confirm that vecmap and fast vectors also do not work on default settings. However, these are not even challenging to the basic coocmap where all cases started to work with less than 30MB of data.

---

For the small but clean Europarl data, tested English to Finnish (fi) and Hungarian (hu). As shown in Figure 2, cooemap started to work at 9MB for hu and 30MB for fi. It finished the transition region when we used all 300MB of en-fi and 100MB of en-hu. vecmap-fasttest has yet to work, agreeing with the results of Søgaard et al. (2018). However, since vecmap-raw using simple SVD worked, it was surprising that fasttext did not. Indeed, decreasing the dimension 100 from 300 enables vecmap-fasttext to start working at 50MB of data for enparl-huparl (vs 8MB for coocmap, Figure 6).
Next result is on English to Chinese (zh), where both are trained on Wikipedia. coocmap had bit more instability vecmap-fasttes also sometimes works with around 1GB of data. The more robust version of coocmap-clip and drop is completely stable, and begin to work with less than 20MB of data.
4.2 Under domain mismatch.
The most difficult case for unsupervised word translation is when data from different domains are used for each language (Søgaard et al., 2018; Marchisio et al., 2020). The original vecmap of Artetxe et al. (2018b) was tested on different types of crawls (WacKy, NewsCrawl, Common Crawl) which did not work in previous methods. Marchisio et al. (2022b) show that vecmap also failed for NewsCrawl to Common Crawl on harder languages. We test on Wikipedia to NewsCrawl and Europarl and see successes enwiki-esnews, enwiki-hunews, and ennews-zhwiki, enparl-eswiki before finally failing on enwiki-fiparl and enwiki-huparl. See Figure 3.
On enwiki-esnews, esnews, where ~ 100MB of data was required to reach 50%, though the basic coocmap becomes unstable and all vecmap or fasttext based methods failed to work at all. However, clip and drop not only stablizes this but enabled it start to work 40MB of data.
On enwiki-hunews, cooemap mostly fails but get 5% aceuracy with above 100MB of data. Impres- sively, clip and drop fully solves this problem well, but even clip has a bit of instability, reaching 50% accuracy at 600MB of data.
On ennews-zhwike coocmap fails completely without clip and drop, and even then requires more data than before. Finally it works and reaching over 50% accuracy around 800MB. In C, we show the data still has higher potential, where truncating to 2000 dimensions enables ennews-zhwiki to work with 300MB or even 100MB of data, though still not reliably.
For more extreme domain mismatch, we test enparl-eswiki. In addition to being small, Europarl contains only parliamentary proceedings which has a distinct style and limited range of topics, to our surprise this also worked with 295MB of enparl, and 500MB from eswiki, reaching final accuracy of 70% suddenly, although basic coocmap also showed no sign of working. All methods failed for enwiki-fiparl and enwiki- hupaar in the range limited by Europarl (300MB for fiparl and 90MB for huparl) with up to 1GB of enwiki, reaching the end of our testing.
5 Analysis: why coocmap outperformed dense vectors
These main results show that high-dimensional coocmap is more data efficient and significantly more robust than the popular oow-dimensional word vectors fasttext/word2vee, which contract prevailing assumptions that vectors are superior and necessary to enable unsupervised translation among other tasks. Ruder et al. (2019) states "word embeddings enables" various interesting cross-lingual phenomena. For unsupervised dictionary induction, landmark papers (Lample et al., 2018a; Artetxe et al., 2018b) needed vectors and even methods that must use a [V] x [V] input constructed these from low dimensional vectors anyway (Alvarez-Melis and Jaakkola, 2018; Marchisio al., 2022a). More generally, the popular textbook Jurafsky and Martin (2023, 6.8) states "dense vectors work better in every NLP task than sparse vectors". Here, provide natural reasons that disadvantage vectors we do not mind having higher dimensions.
The conflicting dimensions of vectors. These properties must hold for vectors to work,
1. Approximate isomorphism: be able to translate by rotation/linear map
2. Denoise: reduce noise and information not robust for the task
3. Retention: keep enough information at given dimension to achieve good performance
By testing the dimension of word vectors in Figure 4, we can see that isomorphism and denoising only holds in low-dimension To see this on enwiki-dewiki where fasttext works well, both vecmap-
7

---

www.kott
eTW KieSTeNS
17-
cocoma--it
coccrap-ils
26-
erw M-tunew
655
C4
0.1
6.1
www.
Figure 3: Accuracy vs. data size with clip and drop. Except for enwiki-eswiki and enwiki-zhwiki (top left, top middle), the rest all have domain mismatch where vecmap- fasttext gets =0.
onwil dowk 201M8
criakd csnews 330V6
eneikiihurews 200vB
s.CP hupe dity Aprapees
0.7
04-
somrrppi ip
05-
ceemmado
655
cerme-do
Calonie sitlacce encmep-fastiot
10
22'
33
Figure 4: Accuracy vs. dimension at 300MB of data. coocmap-clip/drop: trunca(X) by svd, coocmap- fasttext: X normalize((X'X')) for X' RIVixd, vecmap- fasttexx: X' E RRVixd
fasttext and cooemap-fasttext are better than SVD truncation low dimensions. Then accuracy goes to 0 as dimension increases. This is not because vectors fails to retain information, since foocmap--asttexx--lii works if we apply additional clipping. See B for more testing on the effect of dimension and other hyperparameters. Notably, lower dimensions have better isomorphism and denoising, and higher dimensions have better accuracy and retention. This trade off leaves a limited window for fasttext to work well. Still, selecting good dimension is sufficient for fasttext vectors to match the accuracy and data efficiency of coocmap on the easiest cases (en-es, fr, de) and work well enough on dissimilar languages training on the same domain.
Søgaard et al. (2018) notes similar impact of dimensionality themselves, but without exploring enough range that would have solved their enparl-huparl and enparl-fipart tests (with vecmap). Yin and Shen (2018) argues that optimal dimension may exist because of bias-varianc tradeoff. Our experiments show that cooemap keeps improving with higher dimensions, and they may have been misled by relying on linear algebraic properties. The unreliability of isomorphism is also noted by Patra et al. (2019); Marchisio et al. (2022b).
Better denoising in high dimensions. Domain mismatch is where the vectors struggle the most regardless of dimension. In the easiest domain mismatch case of enwiki-esnews of Figure 4, vecmap- fasttext failed whereas cooemap- cooemap-fasttext-de fasttet--clpp worked (though not stably), showing that clip helps even when applied on top of the natural but incidental denoising of vectors.
The association matrix of coocmap normalize(Co" 1/2) is also better overall than other choices such as Mikolov et al. (2013a); Levy and Goldberg (2014); Pennington et al. (2014); Rapp (1995). A, we compared to other full-rank association matrices corresponding to popular vectors and show they also perform worse than cooemap. For instance, the positive pointwise mutual information matrix (PPMI) of Levy and Goldberg (2014) corresponds to word2vec/fasttext. While it can work with
8

---

simple &2 edist without normalize and can reach higher accuracy than the basic coocmap if tested on the same domain, it has lower data efficiency and completely fails on the easiest enwiki-esnews with domain mismatch, showing the same behaviors as fasttext. A more aggressive squishing such as normalize(log(1 + Co)) works robustly too but is less accurate and less data efficient. Impressively, Rapp (1995) also works fully unsupervised with the prescribed lt.
Higher dimensions contain useful information. In all cases, cooemap-clip and coocmap-drop both become more accurate as dimension increases. Under domain mismatch, there is considerable gains above even 1000 dimensions, suggesting that we are losing useful information when going to lower dimensions. Analyzing the association matrices based on which values are clipped, we show in F that higher dimensions retain more world knowledge such as portland-oregon, cbs-60 minutes, molecular-weight, basketball-jordan luisiana-purchase, tokyo-1964 and using low dimensional vectors misses out.
6 Discussions
Better vectors exist. coocmap shows that it is feasible to denoise in high dimensions better than with popular vectors. Thus, applying denoising suitable to task, the size/type of training data first before reducing dimension is more principled ways to train lower dimensional vectors than the current practice of relying on poorly optimizing a supposed objective. In this view, training vectors can just focus on how to best retain information in low dimensions and vectors should just get monotonously better with increasing dimensions and stop at an acceptable loss of information for the task and data.
There is room for improvements for vectors of all dimensions. In Figure 4, fasttext got higher accuracy at impressively low dimensions in favorable conditions whereas SVD on coocmap-drop becomes more accurate with increasing dimension. Without actually constructing the better vectors, it is clear that meaningful improvements are possible: for any given dimension d, there exists rank-d matrices that can achieve a higher accuracy than the max of all rank-d methods already tested in Figure 4.
But why vectors? Here is our opinion why low-dimensional vectors appeared necessary and better beside social reasons. Co-occurrence counts from real data are noisy in very involved ways and violate standard statistical assumptions, however it is obligatory to apply principled statistics such as likelihood based on multinomial or x², whereas vectors take away these bad options and allow less principled but better methods. Statistics says likelihood ratios is the most data efficient: D(s, t) = P,, i) log p or perhaps the Hellinger distance H(s,t) t) = E, llp(ss i) p(t, 1) if we want more robustness. Using raw counts or a term like p plog p (super-linea) in raw counts) always failed when measuring with the fg norm. The likelihood- based method of Ravi and Knight (2011) 1) likely would fail on real data for these reasons. Normalization a common and crucial element of all working methods. The methods of Rapp (1995); Fung (1997): Rapp (1999) were based on statistical principles but with modifications to make normalized measurements. Levy and Goldberg (201 2); Pennington et al. (2014) actually proposed better normalization methods too that would equally apply to co-occurrences (Appendix A).
Giving up on statistics and starting from veemap lead to big improvements. ironic for our attempt to understand unsupervised translation and how exactly vectors enabled it. To prefer more empirical tests on a wider range of operations instead of believing that we can model natural language data with simple statistics might be another underrated lesson from deep learning. In this work, we give up on statistical modelling without giving up on higher dimensions, which makes available these key tools that were sufficient to outperform -in-dimensional vectors: large squeezing operations like sqrt, normalizing against a baseline (subtracting means, divide by marginals), contrasting with others during matching (CSLS), clip and drop. With just normalize and CSL.S, coocmap would have similar accuracy and higher data efficiency compared to fasttext. Clip made co-occurrences more robust than vectors while also increasing accuracy, showing the possibility of denoising in high-dimensions. Finally, drop enabled some access to the information in higher dimensions and lead to noticeably higher accuracy dimension increases. This is actually a more intuitive situation than the prevailing understanding that going to very low dimensions is both more compact and more accurate.
IBM models. cooemap shows that unsupervised word translation would have been possible with the amount of compute and data used to train IBM word alignment models (Brown et al., 1993).

---

While we do test on the exact en-fr Hansards (LDC),, we know it has 1.3 million sentence pairs from the Parliament of Canada. This can be compared to our Europarl en-hu experiments, which has 0.6 million sentence pairs for more distant language pair, showing there may not be much additional information from the sentence alignments. coocmap with drop got around 75% accuracy here. Though the compute requirement of coocmap may have been difficult in '93 by our estimate en-hu with [V] = 5000 probably requires |V|ss * 100 ~ Tera-FLOs, which should have been easy by the early 2000s on computers having Giga-FLOPS.
Limitations. The vocabulary size is a potentially important parameter unexplored in this paper where we used a low 5000 for cleaner/faster experiments. The most expensive step is factorization operations and measuring distances of high dimensional co-occurrences which is O(VV') as opposed to O(d) I22 for vectors, though approximations are available with dimension reduction (see E), Having low vocabulary size may explain why we see little benefit (except in our hardest cases such as C) from the potential denoising effects of truncation. scalable to larger vocabulary and can more easily used by neural models. However, BPE, indexing, truncating vocabulary, multiple window sizes and other alternatives are not explored better use higher dimensions.
now-dimensional vectors is more easily
Speculations. As cross-domain is more challenging than just cross- mooss-l is especially likely that our findings may apply in other cross-domain tasks. So we speculate that if co-occurrences are similarly processed without going to low dimensions, they are likely to perform better on other tasks where more robustness is required than the natural robustness of fow-dimensions. Similarly, neural models may also benefit from intentional denoising like clip and drop.
Beyond simple vectors, more recent language models may also suffer from low dimensions, which might be why they benefit from retrieval (Khandelwal et al., 2020), where low dimensional vectors may lose too much information for the task. Recently, contextual transformer representations superseded non-contextual vectors with more data and more dimensions pushing up the performance across tasks. Following Kaplan et al. (2020), there are important trade-off between dimensions, data and compute. If data is the limiting factor, it is plausible that better ways of using higher dimensions is helpful.
Related work. Rapp (1995) already hinted that there might be enough signal for fully unsuper- vised translation. Fung (1997); Rapp (1999); Haghighi et al. (2008) tested their high dimensional approaches with seed words, but without a full search procedure. Ravi and Knight (2011) succeeded in the case of time expressions and parallel data, but likely used insufficient normalization for harder cases. Mikolov et al. (2013b) noticed the relation between low-dimensional rotations and translation. With good enough vectors, Zhang et al. (2017a); Lample et al. (2018a); Artetxe et al. (2018a) achieved fully unsupervised word translation while showing that seed words do not matter too much as long as the unsupervised method works. There was much follow up on these successes, in particular, even methods that must use an association matrix (Alvarez-Melis and Jaakkola, 2018; Marchisio et al., 2022b) constructed them from low-dimensional vectors.
Broader Impact
For dictionary induction, this work shows it can be done with less data and is more robust to domain mismatch than previously thought. We have working procedure that is more direct, using less compute, making it more accessible for educational purposes. Unsupervised translation means learning from the statistics of data alone and make bland and silly mistakes where outputs should not be relied on unless they can be verified.
A Acknowledgments
Thanks to Mikel Artetxe, Kelly Marchisio, Luke Zettlemoyer, Scott Yih, Freda Shi, Hila Gonen and Tianyi Zhang for helpful discussions.
References
David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-wasserstein alignment of word embedding spaces. In Proc. of EMNLP, pages 1 1881-1890.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-heat baseline for sentence embeddings. In ICLR.
10

---

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations. In Proc. of AAAI.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. A robust self-learning method for fully unsupervised cross cross-lingual mappings of word embeddings. In Proc. of ACL.
Giusepppe Attardi. 2015. Wikiextractor. https://githur http://ataarrr///iiieetttttaaaa
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. Proc of ACL.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the ACL, 5:135-146.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L.. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics.
Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29.
Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In Proc. of ICLR.
Pascale Fung. 1997. Finding terminology translations from non-parallel corpora. Journal of Visual Languages and Computing.
Aria Haghighi, Percy Liang. Taylor Berg- Kirk-KKrkaatrikk, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proc. of ACL.
Zellig S. Harris. 1954. Distributional structure. In Word, volume 10, page 146-162.
Yedid Hoshen and Lior Wolf. 2018. Non-adversarial unsupervised word translation. In Proc. of EMNLP.
Dan Jurafsky and James H. Martin. 2023. Speech and language processing: 3rd Edition. Draft.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, 100/20.0.08361.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. General- ization through memorization: Nearest neighbor language models. In ICLR.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. 2018a. Unsuper- vised machine translation using monolingual corpora only. In Proc. of ICLR.
Guillaume Lample, Alexis Conneau, Marc Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In Proc. of ICLR.
Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. 2015. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In Proc. of ACL, pages 270-280.
Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In NIPS.
Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020. When does unsupervised machine translation work? In Proceedings of WMT, pages 571-583.
Kelly Marchisio, Ali Saad-Eldin, Kevin Duh, Carey Priebe, and Philipp Koehn. 2022a. Bilingual lexicon induction for low-resource languages using graph matching optimal transport. In Proc of EMNLP, pages 2545-2561.

---

Kelly Marchisio, Neha Verma, Kevin Duh, and Philipp Koehn. 2022b. IsoVee: Controlling the relative isomorphism of word embedding spaces. In Proc of EMNLP, pages 6019-6033.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations vector space. arXiv preprint arXiv: 1301.v1301.377 1301.378/.
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013c. Distributed representations words and phrases and their compositionality. In Proc. of Neurips.
Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-top: Simple and effective postprocessing for word representations. In ICLR.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig.
2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces, In Proc. of ACL.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proc of EMNLP.
Reinhard Rapp. 1995. Identifying word translations in non-parallel texts. In Proceedings of the Association for Computational Linguistics.
Reinhard Rapp. 1999. Automatic identification of word translations from unrelated English and German corpora. In Proc. of ACL.
Sujith Ravi and Kevin Knight. 2011. Deciphering foreign language. In Proc of ACL.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569 -31.
Anders Søgaard. 2023. Grounding the vector space of an octopus: Word meaning from raw text. Minds & Machines, 33:33-54,
Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018, On the limitations of unsupervised bilingual dictionary induction. In Proc. of ACL.
Karl Stratos, Michael Collins, and Daniel Hsu. 2015. Model-based word embeddings from decompo- sitions of count matrices. In Proc. of ACL.
Zi Yin and Yuanyuan Shen. 2018. On the dimensionality of word embedding. In Neurips.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017a. Adversarial training for unsuper- vised bilingual lexicon induction. Proc. of ACL.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017b. Earth mover's distance minimiza- tion for unsupervised bilingual lexicon induction. In Proc. of EMNLP.
12

---

A Comparison of association matrices
envik Jonk 50g
CAnION 97s
cocerapeis Resttessa
cocorap Eee339741
cocorap
32-
32*
102
3P
30
3C
32
10
etickl
Kisssosss
concmes
60
0.7
0.7
66-
66-
wile
lnste
09
10.
2000
cala 2012
Cala spe
Figure 5: top: methods from the 90s, bottom: methods from 2010s word2vec and glove.
method
association formula
normalize normalize normalize
edist
coocmap
Co-1/2
E
loglp
log(1 + Co)
8
Rapp (1995)
pla3)
4
p(s)p(e)
Fung (1997)
p(s, i)
log p(s.)
4
13
p(s)p(s)
Levy and Goldberg (2014)
max 10, p(s)p(()
p(a,1)
- -log k
f
E2
/ Church and Hanks (1990)
Pennington et al. (2014)
log(1 + Co) - b. 1T - 1.
l2
82
In Figure 5, we compare to several choices of association method K (Co)) and find all of them to be worse than C/ plus normalize overall, and the two popular methods from 2010s are better than the two methods from 1990s. These methods all perform normalization by factoring out the marginals one way or another.
Besides the association matrices themselves, match them with their best normalization method from normalize, & f2. We report lg rather than normalize for Levy and Goldberg (2014); Penning- ton al. (2014) which proposed their own normalization. In all these cases, normalize gave almost identical results as l,, showing their own normalization was sufficient. normalize corresponds to & edist since it did lg normalization as the last step in (3). We use p(s, i) Co(s, /)/ Ee. Co(s', i') when probability was required.
On the top of Figure 5, we compare Rapp (1995); Fung (1997) from the 90s specifically designed for dictionary induction as well. Impressively, Rapp (1995) also performed well using & distance as prescribed, where it even started to work on enwiki-esnews. Fung (1997) did not work with the prescribed la criteria (not shown) but worked with & as well. We think Rapp (1999) is similar to our modification of Fung (1997) with é,, which contains its most important term. However, using & lead to lower data efficiency and lower accuracy in methods that also work with E2-
On bottom of Figure 5, we compare two popular association matrices from the 2010s. Notably, the positive PMI of (Levy and Goldberg, 2014) corresponds to word2vec/fasttext, and actually has similar performance characteristics as fasttext, where both reach higher accuracy than coocmap on easy cases but has lower data efficiency and fails on enwiki-esnews. Other methods that use normalize on log also worked for enwiki-esnews. Of these, coocmap has the highest accuracy. Levy and Goldberg (2014) achieved higher accuracy while Pennington et al. (2014) worked on enwiki-esnews. The normalization proposed both methods achieved almost identical results with & as with normalize. l also worked on them but clearly worse and has less data efficiency (not shown).
13

---

Pennington al. (2014) did not specify what the biases b should be, instead just leaving them to gradient optimization, we picked them to subtract out the mean of logs whereas PMI would subtract the log of means. 2/2 was also tested by Stratos et al. (2015) and found to be favourable. We also did not find the shifted PPMI to be better than the basic PPMI of Church and Hanks (1990).
B Improving fasttext
eneiki dewii
enpari huparl
0.7
0.7-
type
0.6
mathod
coccmap lastter
coocmedd
0.5
festeechc
0.5-
vecmap
0.4
cooctrap trogg
coocrap- drop
0.3
0.3-
0.2
0.2
15
0.0
0.0
10a
10
10a
30
data size (46)
data size (M95)
enwildi
enwiis hunews
300
0.4
0.6
method type
method Eype
cacermap festhect
coscrrup
coocmap clp
0.3
coocrapp fysteen-cly
vermap foeteett
foctheot
0.4
coucmap drop
coucrrap drop
0.-
0.1-
0.-
0.0-
10a
10F
201
sdow
Figure 6: Improving fasttext with hyperparameter and clipping. Drop was set to min 20 2 resulting in double curve since d 100, 300 were both used.
100
In the main results, we used default parameters, where the important ones were skigram, Ir: 0.05, dim: 300, epoch: 5. analysis section, we improve fasttext further by tuning dimension, learning rate and epoch number, then by using coocmap, clip and drop. The effect of dimension has been specifically explored in Figure 4. Here we show how 100 dimension and 300 dimensions as we vary with the size of data 100-dim tend to be more data efficient whereas 300-dim tend to reach higher performance. The learning rate was slowed as 0.1 (dd5500)) to account for observed instability in higher dimensions. The epoch was increased to 5 x (300/|D))L4 for data size D in MB to run more epoch on smaller data size. We did not observe too much difference between this and default parameters beyond stablizing higher dimensions.
vesmap-fasttext only fully works for the two cases with no domain mismatch enwiki-dewiki and enparl-huparl. In contrast, coocmap-fasttex fasttex show that fasttext contains the information to tackle harder cases of enwiki-esnewe showing that while fasttext retaining the necessary information, it does not have the necessary linear algebraic property for vecmap to work. On enparl Spparl-hupe huparl, vecmap-faat fasttext did work for 300 dimensions whereas coocmap-fasten fasttext--liip did using the same vectors with clipping, achieving better accuracy than recmap-fasttext fasttext 100 dim. On enwiki-esnews only moocmap-fasttext-eoeeee and cooemap-fasttex worked. Finally, all fasttext based methods failed on enwiki-hunews.
always possible that we are still using fasttext poorly, but we covered the main factors and clarified the role of dimension which seems sufficient to explain the observed behaviors.
14

---

C
Clip, drop and truncate
enneAs www.d 303M3
STRRAS shald 3ha4 30193
344
courma
ceeran excerppeR Enp
cocerapess
Figure 7: Accuracy vs. dimension on ennews to zhwiki, on 100MB, 300MB and 1000MB of data.
In this section, we try to push our most difficult case of ennews-zhwiki further by varying hyper- parameters as well as dimensions. In our main results in Figure 3, ennews-zhwiki barely had any transition before reaching high accuracy at 800MB, making us suspect there might be signal before that but the hyperparameter was not optimal. So here we try clip at 1.5% for coopmap-clip-1.55 drop-1.5 instead of I 1% in every other experiment. We also adjust drop to min 20 400 of a constant 20 so that less is dropped for lower dimensions d.
instead
In Figure 7, we get the typical behavior of increasing accuracy with increasing dimension for 1000MB of data all the way to the end. For 100MB and 300MB, this only worked when we also truncate some of the tail end of the SVD. Even here, the maximum accuracy was reached at around 2000 dimensions out of 5000, reaching > 50% accuracy at 300MB of data and 2000 dimensions. It is also remarkable that while clipping more resulted in more stability, the final accuracies are virtually identical when both clip and clip- 1.5 works or when both drop and drop-1.5 works. This is a case where it is helpful to truncate some of the tail of the SVD well, at least for 100MB and 300MB where all failed at full dimensions. However, the accuracy still increased up to 2000 dimensions, much higher than typical vectors for the vocabulary size according to conventional wisdom.
D
More details and ablations
D.1 Effect of initialization
L00INB
conces: 1003MB
coecmep-dimz
Casc ragg fotttot
03-
155
160
drenaaion
dinezaion
Figure 8: Accuracy vs. dimensions enviki-esnews. 1000MB of data. Left: use full coocmap initialization. Top red line is when coocmap-drop uses all 5000 dimensions. Right: use their own initialization.
To rule out that fasttext is only limited by the unsupervised initialization step, Figure 8 shows when all methods use the unsupervised initialization of full rank coocmap (left), vs their own unsupervised initialization (right). Here vecmap-fasttext and fooemap-fasttext-sqrt both worked for a slightly wider range of dimensions but the differences are small. Using the clipped version of initialization also did not make much difference. Overall, varying initializations did not matter too much for unsupervised initializations. On the other hand, for supervised initialization, coocmap tend not be very sensitive once there is enough data. vecmap can be more sticky and is more affected by supervised initialization, possibly because W can reach local minimum.
D.2 Effect of matching method
The matching method was eritical getting cooemap to work, where simply applying the bidirectional matching does not work even on the simplest enwiki-eswiki. It starts to work in the first few iterations,
15

---

but then half of the vocabulary gets matched the word (often but not always [UNK] ) and accuracy go to O. Some hubness mitigation (Dinu et al., 2015) is a must for coocmap. This is not surprising since the indices s, t can make arbitrary matches, whereas the vector space cannot make arbitrary matches from rigid rotation.
We tried naive and greedy mutual matching method that also solved the problem, but CSLS is better, simpler and is more similar to prior works. A proper matching methods such as Hungarian algorithm softer matching method already tested bilingual dictionary induction (Marchisio et al., 2022a) should work as well.
we used fasttext since it is more standard for this problem, and we also checked that the subwords information made no difference by turning them off in fasttext hyperparameters. This is reassuring for the findings to apply word2vec as well.
Subwords.
E
From vectors to association matrices
We derive more precise relationship between cooemap and veemap using the notations of Section 3.
Let the d-dimensional vectors be X, Z E R such that they are whitened X = ZT la- Then for coocmap, the association matrices is KYY) = (YYT))/2 = YYT. We can show that for dot product edist, and permutations s, t,
cdist (X)) s). [( A) = cdist (W,, Z)
(5)
where the LHS the distance function in cooemap of Alg and the RHS the distance function in vecmap of Alg 2. To see this,
edist (X)( (Z) 4)
[(X)[,s) K( T)T
(
-XX[s]] 2[1]2T
if X Z(t), then W XXXx since (X[s] = 1, W = [
-x
ccdis((W,,2)
We can try relaxing the whitening assumption. w.l.o.g., let U,E,, 2 U,Σ₂ where UTU, Ia- This can be obtained via SVD on any X' U, U,E, VT so that the association matrix is not affected, i.e. K(X') (X' 1/2 (XXT)1/2 U2 U... In this case, W [U] is the mapping that would make the two sides equal.
However, if we try least square solve as before, we get W , The Procrustes solution for [[s] svd-norm takes the SVD but set all singular values to 1. Neither will make the two sides equal
Ztt] (or U1 [s[s|E, U2 {2) is W = svd- [vd-norm(_[Uils] {
exactly but these have some similarities.
F
What is in higher dimensions?
Although the increasing accuracy, the absolute results and the consistency when we keep higher dimensions suggest that there are useful signals, it could always be that we are using lower dimensions poorly or benefting from other unknown effects. In this section, we compare the full-rank matrix and the low-rank matrix to see what exactly is the difference. If the information lost by low rank seems useful robust, then we can be more confident in retaining this information. Indeed, the low rank version seem to lose or de-emphasize lot of world knowledge, specific phrases and the like while making the association matrix more like similarity matrix. In this section, compare the full-rank matrix and the low-rank matrix to see what exactly is the difference. If the information lost by low rank looks useful robust, then low rank version seem to lose or de-emphasize a lot of world knowledge, specific phrases and the like while making the association matrix more like similarity matrix.
be more confident in retaining this information. Indeed, the
We construct the full dimensional matrix, apply normalize and drop. Then we use the thresholds established by clip. If an entry needs to be clipped in the full-rank matrix, but somehow is already
16

---

reduced and longer need be clipped in the lower-rank matrix, record this full-rank+ in Table 9. Conversely, if low-rank instead increased an entry so that should be elipped, we note it as 300-dim+.
From these examples, the differences are clear. The full dimensional matrix contain more world knowledge and otherwise favor dissimilar words that are highly related: richard-nixon (famous person), portland-ores -orgoon maine (geography), error-margin, molecular-weight (scientife term), tokyo- 1964 (olympic), basketball-jore jordan james (famous players), cbs-60 minutes (show CBS), louisiana- parish purchase (famous event).
In contrast, the 300-dim+ favors identical or similar words like state-state, age-age, who-whom, truth-ruee, tokye-pyeongchang. chs-fox, family-householde
making the association matrix more like the similarity matrix. This is also intuitive from SVD as well: if the association matrix is USVT, then the similarity matrix is U VT whereas low dimension is U V with both S and S = (ru emphasizing higher singular values.
We also tested full-rank 1000-dim, where the effects are similar the full-rank vs 300-dim shown, but more subtle and harder to present clearly. When we tested drop we found drop tend to reduce associations between pairs of numbers, words and punctuations and the like, whereas it increased associations between article and words. It is not clear if this is good or bad.
G Example predictions
We include some predictions for extra information. The P@ I is quite generous to small vocabulary V, since does not evaluate on entries if the source not in Vi, no targets are in the V2, treating the evaluation as out-of-vocabulary instead. So we show the number of correct predictions and the source overlap with the reference dictionary, both out 5000. Results are shown in Table 10.
17

---

source
full-rank+
source
full-rank+
full-rank+
300-dim+
full-rank+
300-dim+
NewsCrawl
Wikipedia
source: tokyo
source: truth
richard nixon
families
residing
NewsCrawi
NewsCrawi
cooper anderson
called
japanese
winter
speak
true
anderson cooper
aspect
ratio
prosecutors
south
commission
report contributed
margin
electric
china
Ihan
at
least walter
scott
metropolitan
korea
be
no longer
another
detention
pyeongchang
uncomfortable
story
tham rather oregon portland
Wikipedia
Wikipedia
this
earlier scott
walter
bay
chinese
tell
god
dean wells
unless
otherwise
1964
china
commission
view
kyle walker
usa
loday
2020
koreu
table than
crazy rich
jones
indiana
ward
areas
functional
universe
behind scones
louisiana
parish
stock
san
scarch
which
kyle
british
columbia
rose
politically
correct
water
%
source: basketball
thomas
cooper
alice
source: cbs
NewsCrawl
graham
billy
maryland baltimore
NewsCrawl
operations
club
why
face
fox
corruption
rugby
detroit
lions
illinois chicago
60
night
division
nil
correct
politically
lord rings
nation
channel
courts
clubs
aaron
rodgers
jumes bond
boston telling
magic golf
2000
russell
siege laid
minutes
abc
james
ball
star
wars
56). persons
Wikipedia
Wikipedia
light
shed
molccular weight
60
bbc
nba
club
town
hall
then since
minutes
disney
operations
stadium
almost
vertainly bob hope
columbia
cuble country ice
holdywood
reporter
even walter live
memorial
rugby
texas
austin
coast
guard
ruther
shows
college
clubs
cash
flow
final
fantasy
Figure 9: Left: word pairs favored by the full rank matrix (full-rank+) on NewsCrawl or Wikipedia sorted by the amount of difference consists of identical words with the rest mostly synonyms. Right: same comparison but for selected words, also sorted by the difference.
all pairs of words. 300-dim+ not shown since 72/top 100
18

---

ennews swift
zhwiki
enwiki victims
esnews
enwiki looking
dewiki
4458
3484
víctimas
2732
ergaben
947
significant
96
2727
somewhat
bastante
3056
deputy 1805
stellvertreter
2179
strength
23 _
682
star centers
estrella
1977
1977
2972
letters
f6
2847
centros
337
september
september
2401
coverage
4176
theology
sindicatos
4669 strict
streng
733
idea
12 02
868 historic
históricos
2248 sector
hereich
4260
assist
160 80
451 research
investigaciones
414 came
kam
2803
properties
3548 tell
comprobar
105
while
hingegen
4542
circuit
2375 script
estuvieron
county
county
4191
saint
3570 elevation
minima
702
pacific
hewohnem
612
recently
3021 800
600
248
house product
baus
3360 maria
SB
765 forms
formas
1178
produkt
1846
surprise
4575 delayed
adelantado
164
second
zweite
713
copyright
1907
contained
enthiclt
2015
carrying
19 90
4058 accurate
precisa
3208
easy
schwer
3843
suspects
B 9
3708 technologies
tecnologías
3200
tone
ton
1536
dropped
7
854 elements
elementos
4578 consensus
zustande
1014
largest
3187 abbey
estás
television
fernschen
2672
communica
189 57.
1483
mountains
siera
1716 75
75
4934
masters
2F
4708
southwestern
básico
4489 handed
übergeben
989 words
19
1094
problems
problemas
2866 electrical
elektrische
4400 pan
3813 compete
competir
2976 proposal
vorschlug
3289 maximum
ELE
934 remains
sido
4788 beta
1620 ceo
1642 crown
corona
691 food
nahrung
821 offer
ftt
3045 harbor
sentirse
251 power
macht
2611
4274 phone
teléfono
konnte
1520 individual
A
3626
subraya
286 late
späten
801 cancer
302 german
alemán
290 park
park
4003
11
133 early
naranja
3903 en
en
319 always
65
4955 provisions
vigente
808 source
quelle
4424 backs
60
3366 fruit
aceite
1050
kosten
1345 modern
kt.ft
22499 engines
motores
1803 99
99
4835 spell
19
4508 trump
trump
2179 stand
stehen
4737 loud
2669 forming
formar
3926 pitch ton
3894
metres
2R
personal
personales
4449 populated
besiedelt
4472 flood
1810 identity
identidad
3979 hunt
jagd
4648
attract
782 strong
fuerte
4909 wore
trugen
3297 lowest
936 recent
recientes
2510 seems
scheint
$39 50
always
siempre
3978
bruder
1083
build
72 into
2325
operate
betreiben
correct
1411
correct
2024
Correct
2424
source
3341
source
4351
source
4401
P@
66.30%
P@
71.60%
P@
77.05%
Figure 10: 40 random samples from all predictions. Source indices are included to reflect frequency where smaller means more frequent in the source language (left columns).
19


Reviews:
[{"review_id": "t8ujpX1We4", "reviewer": "Reviewer_VTy1", "summary": "In this paper, the authors target to solve unsupervised word translation, also called lexicon or dictionary induction, using their proposed method, coocmap, a simplified version of the conventional way, vecmap. Different from vecmap, coocmap can estimate word mapping without using rotation weights for row vectors. Instead, coocmap uses an association matrix to represent source and target word relations. Thus, coocmap can estimate source and target word mappings by rearranging the columns of the association matrix to reflect the updated word mappings during training time. Experimental results on language pairs from English to Spanish, French, German, Hungarian, Finnish, and Chinese show that coocmap outperforms vecmap, especially when the data size or dimension size increase.", "questions": "- In the evaluation, why didn't you compare the performance of your coocmap with the reported scores of conventional methods? You can use Semeval 2017 [2] data for this purpose, similar to the paper of MUSE dictionaries [1] you used.\n- Could you expand your approach to using subwords? This is because current natural language generation tasks, including machine translation, heavily rely on subwords rather than words. This part relates to the contribution of your work to downstream tasks.\n- In the current draft, you use MB to show the size of the data. How did you calculate them? The data you used is compressed or not?\n- It's difficult to estimate the computational complexity or space of both coocmap and vecmap by MB of the used data. Showing the vocabulary size is more helpful for readers to understand.\n\n[2] Camacho-Collados, J., Pilehvar, M. T., Collier, N., & Navigli, R. (2017, August). Semeval-2017 task 2: Multilingual and cross-lingual semantic word similarity. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017) (pp. 15-26). (https://aclanthology.org/S17-2002/)", "limitations": "You need to include the treatment of the unseen words in the limitation part of your paper. Furthermore, you need to refer to the fact that current translation models commonly use subwords, not words.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method, coocmap, outperformed the conventional approach, vecmap, in word translation accuracy on language pairs from English to Spanish, French, German, Hungarian, Finnish, and Chinese.\n- Coocmap does not require a rotation matrix for projecting word embeddings to the ones of other languages; thus, it's simple.", "weaknesses": "- Not using the rotation matrix means coocmap cannot handle unseen words because their representation is not included in the association matrix of coocmap. Thus, the versatility of coocmap is less wide compared with vecmap.\n- Many translation tasks require translations for texts rather than word translation. Thus, investigation of downstream tasks, especially for machine translation, is necessary to claim the advantage of coocmap even though it's not conducted in the paper.\n- The motivation for using the unsupervised method needs to be clarified. In practice, we can use the MUSE [1] dictionaries you used to evaluate your approach, to train word translation models in a supervised manner.\n\n[1] Lample, G., Conneau, A., Ranzato, M. A., Denoyer, L., & Jégou, H. (2018, February). Word translation without parallel data. In International Conference on Learning Representations. (paper: https://openreview.net/forum?id=H196sainb, code:https://github.com/facebookresearch/MUSE) ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review with many concrete points that may help us avoid possible confusions. We first address the weaknesses and then the questions. We emphasize that the most substantial proposed weakness about unseen word is wrong.\n\n> Not using the rotation matrix means coocmap cannot handle unseen words... versatility  less wide compared with vecmap\n> You need to include the treatment of the unseen words in the limitation part of your paper. \n\nThere is no difference in the treatment of unseen words. For words unseen during matching process but seen in the corpus, you can use the occurrences of the unseen words with seen words to form the association matrix. The case of adding a new corpus containing new words can be treated similarly. For words unseen in any corpus during training, we also cannot get vectors for them and any rotation matrix is also useless.\n\n> In practice, we can use the MUSE [1] dictionaries you used to evaluate your approach, to train word translation models in a supervised manner.\n\nIt has been shown in previous work that supervised translation can work to some extend in the 90s and 00s, but it is only after vectors/pretraining that unsupervised word translation started working in the late 10s. So the community inferred that there is some magic about low-dimensional representations that must be necessary. Thus it is only interesting if we show that the full unsupervised self-learning loop can work in high-dimensions.\n\nIn fact, table 1 of your citation [1] shows that supervised vs unsupervised does not make a big difference as long as the self-learning loop gets started unsupervised. Increasingly, as LLMs are shown to perform translation unsupervised, it is increasingly difficult to argue that supervised is always the better choice in practice.\n\n> investigation of downstream tasks, especially for machine translation, is necessary to claim the advantage of coocmap even though it's not conducted in the paper.\n\nThe main contribution of coocmap is conceptual by showing that low-dimensions is not necessary and denoising can be effectively performed in high-dimension. Note you can use the same objection against the cited MUSE, vecmap, word2vec etc and the evaluation that you cited. While it would be nice if we can also get results on translating sentences or documents, we thought translating words unsupervised is an interesting enough problem that has a very intuitive evaluation.\n\nThanks for the questions.\n\n> In the evaluation, why didn't you compare the performance of your coocmap with the reported scores of conventional methods? You can use Semeval 2017 [2] data for this purpose\n\nThe included evaluation were deemed enough to support the main points claimed by the paper that you do not need vectors and keeping high dimensions is useful. We do not claim that this method is better than reported scores of \"conventional methods\" when the amount of data is big enough that everything works. \n\n> Could you expand your approach to using subwords? \n> Furthermore, you need to refer to the fact that current translation models commonly use subwords, not words.\n\nYes, you can build the subword association matrix as well and even more generally use featurized co-occurences that may include subwords as features. Regardless of how the latest NLP model is tokenized, the word is basic concept recognized by almost everyone, is a better unit for translation than subwords, and used by previous works / evaluations. While subwords represents a good practical trade off point for transformers, it does not render evaluating on words invalid and would only be a complication for this work.\n\n> In the current draft, you use MB to show the size of the data. How did you calculate them? The data you used is compressed or not? Showing the vocabulary size is more helpful for readers to understand.\n\nMB is megabytes of uncompressed text data in utc-8 encoding. The paper clearly states that any data is decompressed if it came compressed. The caption of table 1 also states how to convert the MB to token counts at roughly 0.2 million tokens per MB. It can be counted by various unix and python tools such as 'f.read(1000)' to read 1000 bytes of data. As you pointed out, the token count depends on how you tokenized, thus we picked MB but would not object to token count.\n\n> It's difficult to estimate the computational complexity or space of both coocmap and vecmap by MB of the used data. \n\nIn both methods, estimating occurrences or vectors is linear in the size of the data. The remaining matching process is a function of the vocabulary size and independent of the data size. In practice, the data size dependent part of coocmap is very fast, just requiring a counting pass over the data whereas estimating vectors tend to run much slower.\n"}, {"review_id": "o9AHT4Vlab", "reviewer": "Reviewer_ZKax", "summary": "This paper proposes coocmap, a method for unsupervised word translation that uses high-dimensional co-occurrence statistics instead of low-dimensional vectors. The authors show that relying on low-dimensional vectors can lead to suboptimal denoising methods and overlook useful world knowledge in high dimensions. The authors demonstrate that unsupervised translation can be accomplished with a smaller amount of data and in a wider range of scenarios than previously believed. They also suggest that co-occurrence-based methods may outperform low-dimensional vectors in other tasks.", "questions": "1.\tIn Figure2, there is a sharp surge of performance for vecmap-fasttext on enwiki-frwiki and enwiki-eswiki. Can you explain this?", "limitations": "see weakness", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper successfully demonstrates the effectiveness of high-dimensional co-occurrence statistics and provides empirical evidence that unsupervised translation using only co-occurrence statistics is feasible.\n2.\tThe authors provide a detailed comparison with vecmap and present coocmap in a clear and easy-to-follow manner.\n3.\tThe paper showcases the trend of retrieval accuracy during the training process on the data size, which is valuable for understanding the training dynamics of word translation capacity.\n", "weaknesses": "1.\tThe experiment setup in the paper is somewhat ambiguous, as the authors state that the experiment was conducted on the top 5000 most common words in each language using the MUSE dictionary, but do not provide a clear definition of what they mean by \"most common.\" Additionally, the MUSE dictionary contains a significant number of \"identical translation\" pairs, where the source word and corresponding target word are the same. It is unclear how these types of translation pairs were processed in the experiment.\n2.\tThe method is only compared with vecmap. Maybe more baseline methods such as LNMAP[1], FIPP[2] and [3] should be included for a comprehensive understanding of model performance. \n3.\tThe paper could benefit from a more comprehensive discussion of prior works in the field, which would help readers contextualize the authors' contributions. It is unclear whether this is the first paper to use co-occurrence statistics to induce an unsupervised word translation dictionary, and it would be helpful to know if there are any previous works in this area.\n\n[1]LNMap: Departures from isomorphic assumption in bilingual lexicon induction through nonlinear mapping in latent space.\n[2]FILTERED INNER PRODUCT PROJECTION FOR CROSSLINGUAL EMBEDDING ALIGNMENT\n[3] Improving Word Translation via Two-Stage Contrastive Learning\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review, we are glad to hear that you found the paper easy to follow. In particular, we claim that we are the first to do unsupervised word translation with just co-occurrences and there are several difficulties with doing this which we identified in the paper. More details in the answer to the particular question, please do reconsider your score as this is an important claim.\n\n> The experiment setup in the paper is somewhat ambiguous, as the authors state that the experiment was conducted on the top 5000 most common words in each language using the MUSE dictionary, but do not provide a clear definition of what they mean by \"most common.\"\n\nThe most common words is based on word frequency in the corpus and returned by the WordLevel tokenizer. In python, `Counter(corpus).most_common(5000)` gets the 5000 most common words when `corpus` is a list of words. For example, the most common word in `corpus=['cat', 'cat', 'dog']` would be `{'cat'}`.  We will add \"by word frequency\" to further clarify.\n\n> Additionally, the MUSE dictionary contains a significant number of \"identical translation\" pairs\n\nYes indeed identical translation is a known issue with the evaluation set. We avoided the worst cases for Chinese / English by filtering out all Latin characters from the Chinese corpus, so there is at least no identical words for the en-zh pair. For other languages where words may actually be identical, we ignore this issue but track how many identical pairs were matched in the experiments. This and other problems with the benchmark is only important if we claim that small improvements on MUSE are an important contribution, which we do not. On the other hand, the MUSE evaluation is good enough to establish if unsupervised translation is happening at all and at how much data is needed, which is our main evaluation. \n\n> The method is only compared with vecmap... more baseline methods such as LNMAP[1], FIPP[2] and [3] should be included \n\nBecause establishing acc(coocmap) > acc(other method) is not a main point of the paper. All of the cited methods are also vector space methods that depends more on which vectors were used and on what data they are trained on. Those methods also compared with vecmap in their own papers. We share your concerns that MUSE/other word translation evaluations should not be taken too seriously, and we do not use MUSE to claim that method X has slightly higher accuracy than method Y. Instead we use MUSE to test which method works at all and how much data was needed. Note that we do not even claim that acc(coocmap) > acc(vecmap), as vecmap with enough data outperformed the basic coocmap as shown in Figure 2. It is only with \"drop\" and high dimensions that coocmap has higher accuracy than vecmap where drop changes the inputs but not the method.\n\nWe compared to other normalization methods and other word vectors in the appendix, which was interesting but did not make the main paper.\n\n> It is unclear whether this is the first paper to use co-occurrence statistics to induce an unsupervised word translation dictionary, and it would be helpful to know if there are any previous works in this area.\n\nThanks for the feedback. As far as we know, this is indeed the first paper to use occurrences to do unsupervised word translation successfully. We cited works on previous attempts that ultimately did not work unsupervised on general data. These previous works are cited prominently in the first sentence of introduction (Rapp 1995, Ravi and Knight 2011) and with more details in the discussions section and appendix. Rapp 1995 suggests that this might be possible but did not have the search methods. Ravi and Knight actually used parallel data in narrow domain like time expression and subtitles for their method to work. Their method also does not include the normalization and relative measurements that were crucial for coocmap to work.\n\nIn fact, we probably could not do this successfully without knowing that it is possible and without some of the advances from vector based methods such as normalization and relative measurements, but vectors themselves were not necessary. \n\n> In Figure2, there is a sharp surge of performance for vecmap-fasttext on enwiki-frwiki and enwiki-eswiki. Can you explain this?\n\nAll methods need a sufficient amount of data to work and the transition is usually fairly sharp. The intuition is once you can figure out an initial amount of word translations, then other frequent enough words should be easy to translate based on context. \n\n\n"}, {"review_id": "H1ktx74L63", "reviewer": "Reviewer_RpPs", "summary": "This paper introduces coocmap, an approach for unsupervised word translation that uses high-dimensional co-occurrence statistics instead of lower-dimensional word embeddings. The approach is generally analogous to vecmap, alternating between distance computations and a matching phase. Coocmap is often more data efficient than other methods, and its performance may be improved by regularizing the co-occurence matrix (normalization, value clipping, dropping large singular vectors).", "questions": "Why use 50% as a threshold for \"works\"?\n\nGiven that you use high-dimensional co-occurrence statistics, what are the compute and memory requirements compared to approaches that use low-dimensional vectors?\n\n[L45] retain -> retaining\n\n[L48] have -> having\n\n[L286] Fragment \"Actually showing the same behaviors as fasttext.\"\n\n[L292] retain\n\n[L303] modeling (or modelling, or \"to model\")\n\n[L336] focuses", "limitations": "Yes, the authors discussed limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed approach goes against the conventional wisdom by using high-dimensional co-occurrence vectors instead of word embeddings. The method is generally more data efficient than previous work, which may be surprising.\n\nThe paper describes strategies to improve upon using the raw co-occurrence matrices directly.\n\nThe experiments cover multiple languages (although English is part of every pair) and domains.", "weaknesses": "Given that unsupervised word translation has been introduced many years ago, but arguably has limited practical applications, the authors should more clearly motivate their work.\n\nIn addition to data efficiency, final accuracy performance should also be discussed (although we can read it from the figures).\n\nIt might have been interesting to analyze how well the approach works based on word frequency (and with vocabulary sizes beyond 5000).\n\nThe paper can be difficult to read at times. vecmap should arguably be described before coocmap, although presenting them in parallel may still be acceptable.\n\nThe paper slightly exceeds the 9-page limit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review and concrete corrections. We will make the corrections in the final version.\n\n> Given that unsupervised word translation has been introduced many years ago, but arguably has limited practical applications, the authors should more clearly motivate their work.\n\nThe particular MUSE evaluation was indeed introduced \"many years ago\" and the underlying task of word translation has been around for even more years, which shows how intuitive the task is. Our main contribution is to show that unsupervised translation is also possible without word vectors and that the data requirement is surprisingly little using a simple method.\n\nMore generally, as LLMs clearly demonstrated unsupervised abilities, future MT systems are likely to use more unsupervised data. The path from research to practical application may take time and immediate practical application should not be a requirement for research.\n\n> In addition to data efficiency, final accuracy performance should also be discussed (although we can read it from the figures).\n\nThe final accuracy was judged not to be unreliable as several previous work warned against taking this benchmark too seriously. On the other hand, MUSE is absolutely good enough to tell if the unsupervised method is successful at all.\n\n\n> It might have been interesting to analyze how well the approach works based on word frequency (and with vocabulary sizes beyond 5000).\n\nYes, though we needed a cutoff to develop the method quickly and keep it simple. An incremental approach where the most common words is solved first then extending the search to larger vocabulary is fairly promising but still introduces complications in search strategy and implementation. This point is stated in limitations and we will emphasize more.\n\n> The paper slightly exceeds the 9-page limit.\n\nThanks for pointing this out and for overlooking this mistake, we followed instructions that \"additional pages containing only broader impact statement and references are allowed\", which seems outdated. Will fix for the final version.\n\n> Why use 50% as a threshold for \"works\"?\n\nThis is just an arbitrary threshold so we can measure the data requirement. Since the accuracy is never 100%, and are binary in that they are either near 0% or is eventually greater than 60%. It would have been fine to pick anywhere between 10% and 60% to establish the comparison.  One analogy supporting the choice of 50% is the half-life of an exponential decay, which is admittedly more precise than our use case. \n\n> Given that you use high-dimensional co-occurrence statistics, what are the compute and memory requirements compared to approaches that use low-dimensional vectors?\n\nThis is discussed in limitations for compute as a function of vocabulary size V, which favors d-dimensional vectors by a factor of V/d. For memory, both methods are O(V^2). In term of data size scaling, both methods are constant.\n"}, {"review_id": "9JHQyP9QNR", "reviewer": "Reviewer_AbFD", "summary": "This paper proposes using high-dimensional co-occurrence statistics for unsupervised word translation rather than relying on low-dimensional vector embeddings. They propose using coocmap (uses association matrix of cooccurrences) and combining this with regularization operations (clip/drop) that eliminate noise. The goal is to achieve good performance on this task more efficiently (less data+compute) and good cross-domain performance.\n\nExperiments are performed on 7 languages and 3 domains, measuring precision@1 on 5K most frequent words against the MUSE dictionary. The baseline uses fasttext vectors. Coocmap+variants outperform fasttext vectors across languages that are not very similar and across domains in some cases using an order of magnitude less data. Analysis shows that high-dimensional co-occurrence data is more robust than its low-dimensional counterparts.\n", "questions": "Could some of the headroom in the vectors be covered by distilling from coocmap to low-dimensional vectors? i.e. is it an optimization issue or a capacity issue?", "limitations": "See above for concerns regarding transferring to other tasks and being able to scale easily.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. This work is interesting and identifies that for the task of unsupervised word translation the raw learning signal generalizes better than compressing into low-dimensional vectors which are lossy.\n2. The domain mismatch experiments perhaps highlight this point well by relying on more fine-grained information that may be lost in the embeddings.\n3. The proposed denoising functions are clever and simple.\n", "weaknesses": "The current draft is at times not very articulate and it can take a few passes to understand the intended point. Some revisions might help!\n\nIs the following an accurate characterization of the work: low-dimensional representations are not expressive enough for this task so we start by overfitting with cooccurrences and use some heuristics to regularize?  \nIf yes, then does this feel more like gaming the task? And would that make the claim about being able to transfer to other monolingual tasks unfair? Doesn’t seem too convincing yet.\n\nScaling does still seem like a challenge: would switching to BPE preserve the multilingual distributional hypothesis?\n\nNit: including low-resource languages in the experiments might make the paper stronger. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review and feedback. We are glad that you liked the content of the paper and we will try harder to make the paper more articulate in future revisions.\n\n> Is the following an accurate characterization of the work: low-dimensional representations are not expressive enough for this task so we start by overfitting with cooccurrences and use some heuristics to regularize?\n\n> If yes, then does this feel more like gaming the task? And would that make the claim about being able to transfer to other monolingual tasks unfair? Doesn’t seem too convincing yet.\n\nThis is not accurate. As there is no training data to fit but only unlabeled corpus, there is no overfitting here and no regularization. As for expressivity, low-dimensions representations is how previous work solved unsupervised word translation, showing the power and ease-of-use of low-dimensional vectors. However the community then draws the wrong conclusion from this success, concluding that vectors are also better in every way and there is only noise in high-dimensions to be removed. We show that a few simple processing steps actually allow us to use the full high-dimensional co-occurrences, leading to similar and then better results than low-dimensional vectors.\n\nNo, we could not game the task and do not see a good way to game it as the task is fully unsupervised. \nSpeculations were clearly labelled and our reasoning is this:\n1) naively using high dimension may indeed lead to conclusions like in [1] because no denoising was used at all\n2) here is a way to use the higher dimensional information for unsupervised MT so that higher dimension is helpful\nThus we speculate that the actual situation is that the necessary but simple denoising operations were not used in previous work, which leads to the conclusions in [1]. This proposed explanation is actually much more intuitive than the current \"vectors make all NLP tasks better\". The reviewer can draw their own conclusions about this, but the author is willing to bet on it. We will try to make this more clear.\n\n[1]: Baroni et al. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors \n\n> Scaling does still seem like a challenge: would switching to BPE preserve the multilingual distributional hypothesis?\n\nScaling coocmap to more data is easier than with word vectors. Scaling to more word types is more challenging than using word vectors, but some incremental approach (limiting the search space to words of similar frequency) and approximations (including truncation and vectorization) can be used.\n\nBPE with large enough vocabulary should preserve at least some of the multilingual distributional hypothesis but probably not as well as words, which are betters unit for translation than subwords. BPE can be combined into coocmap by featurization.\n\n> Could some of the headroom in the vectors be covered by distilling from coocmap to low-dimensional vectors? i.e. is it an optimization issue or a capacity issue?\n\nExperiments included in the paper show the gap is probably more of a \"capacity\" issue, in the sense that low-dimensional vectors lost useful information. In the experiment of coocmap-fasttext vs coocmap in figure 4, the same optimization method was used, only their \"capacity\" differs.  \n\nNot sure how distilling applies here exactly, but word vectors are probably doing as good a job as they can given their low dimensions. We showed their superiority over SVDs at the same dimension."}, {"review_id": "jOJSLQS6so", "reviewer": "Reviewer_LMyn", "summary": "The authors argue that relying solely on low-dimensional vectors misses out on better denoising methods and valuable world knowledge present in high dimensions. Therefore, they propose coocmap, a method that can exploit high-dimensional co-occurrence counts. Through extensive experiments, the authors show coocmap works very well under different data sizes and data from various domains. ", "questions": "- Line 12: \"similar data\", the author should specify similar in terms of what?\n\n- As vocabulary size is an important hyperparameter of the method. The authors should mention it in the main content (probably in the First paragraph in Section 4). The authors are also encouraged to explore the influence of vocabulary size on performance.\n\n- The authors should provide more intuition on why clip and drop are very helpful for harder cases, e.g., en-zh. As the vanilla coocmap is not stable on en-zh.\n\n- Line 194: \"NewsCrawl 2019.es and ...\" -> remove \"and\"\n\n- Figure 2: It would be great if the authors could say the upper 3 figures are similar languages while the lower three are less similar languages in the caption.\n\n- I have a more general question: the study itself is interesting and demonstrates how a simple higher-dimensional co-occurrence matrix could largely help unsupervised word translation. However, in the era of LLMs, when and where can the method be possibly applied? ", "limitations": "The authors discussed the limitation and broader impact of the paper. It is encouraged to include some additional experiments to explore the influence of vocabulary size.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-presented and easy to follow.\n- The figures are illustrative and informative.\n- The authors conduct extensive experiments on different language pairs, including similar and dissimilar languages. Besides, the authors also explore the influence of data sizes and domain mismatch.\n- The method proposed, coocmap, is straightforward but effective. Compared with some baselines, the method seems to work surprisingly well on small data sizes.\n", "weaknesses": "There is no apparent weakness in this paper. However, there are some questions/suggestions (see below). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful review and corrections, which will help us improve the revision.\n\n> Line 12: \"similar data\", the author should specify similar in terms of what?\n\nWe will specify similar in domains.\n\n> The authors should provide more intuition on why clip and drop are very helpful for harder cases, e.g., en-zh. As the vanilla coocmap is not stable on en-zh.\n\nThanks for the suggestion, we will discuss more. Clip can be motivated by robust statistics and some clipping happens automatically for SVD truncation. We also motivated clip intuitively in the section it was introduced but not specifically for harder cases. As for drop, we provided two citations that did it to a lesser extend and we admittedly do not understand drop as well as clip. One additional intuition is that drop already happened for free in word vectors.  \n\n> As vocabulary size is an important hyperparameter of the method. The authors should mention it in the main content (probably in the First paragraph in Section 4). The authors are also encouraged to explore the influence of vocabulary size on performance.\n\nWe will add a summary of the main points about vocabulary size to the first paragraph of Section 4 in addition to reference to limitations.\n\n> the study itself is interesting and demonstrates how a simple higher-dimensional co-occurrence matrix could largely help unsupervised word translation. However, in the era of LLMs, when and where can the method be possibly applied?\n\nIt is easier to use the findings than the method directly. Retrieval augmented LMs and knn LMs are perhaps two places where the findings may apply. Transformer LMs still have a low dimensional embedding layer which is typically a few thousand dimensions for each subword. Previous work have shown that knn and retrieval can improve the perplexity, but did not explain why. Our results suggest that the information lost in dimension reduction is probably why retrieval can be beneficial. Note that retrieval is just indexing for higher-order co-occurrences and raise the possibility that simply having the full dimensional co-occurence may also be helpful and capture information not already captured by transformer models. \n\nIt would be pretty interesting to see if the drop and clip techniques are applicable in LLMs directly since we show that even low-dimensional vectors can benefit from clip, which suggests that other activations that approximates co-occurences may also benefit from clipping. On the other hand, drop was needed to actually get the improvement with higher dimensions, which may also be helpful to projects attempting to augment LLMs with a sparse high dimensional object.  "}]

Author rebuttal:
We thank all reviewers for the insightful and helpful reviews. We will aim to improve the draft using the feedback and make it more clear. A few common points are emphasized here and other points and questions are addressed in individual responses.

1. We are the first to use just co-occurrences to achieve fully unsupervised word translation. To do this, we needed the normalization and relative measurements which was developed for vector methods but turned out to be even more crucial for co-occurrence methods. Interestingly, it turns out that these methods rather than vectors were the key. In addition to normalization and relative measurements, which enabled co-occurences to work at all but slightly less accurate than vectors, clip and drop was needed to really make higher dimensional vectors more robust and more accurate than well-trained low-dimensional vectors on exactly the same data.   

2. The main conceptual contribution is to change our understanding from "dense vectors work better in every NLP task than sparse vectors" (Jurafsky and Martin, 2023,6.8) to "there is useful information in higher dimensions" accessible after applying simple techniques such as normalization, relative measurement plus drop and clip. We back up this new view with detailed analysis and experiments on dimensions, which is in retrospect more plausible/intuitive than the prevailing view of the community. 

On the other hand, we do not consider achieving slightly higher accuracy as a main contribution nor very interesting. In fact, figure 2 shows that vecmap often has slightly better accuracy than the basic coocmap with enough data. We do not think the word translation task is sensitive enough such that gaining a few percentage more accuracy is important/interesting. We are aware of its limitations both pointed out by some reviewers and by previous work (identical words, lots of proper nouns). On the other hand, we are still happy to use word translation as the evaluation because it is very intuitive and it is absolutely good enough for testing if unsupervised translation is succeeding at all. These known issues are not relevant if we only focus on the binary phenomenal and data efficiency instead of a few percentage difference in the accuracy.

Meta-review: The paper proposes a method that uses high-dimensional co-occurrence counts or their lower-dimensional approximations for unsupervised machine translation. The evaluation results showed that the proposed approach outperformed conventional approaches. The paper can be further strengthened with the inclusion of better motivation and clarity in the description of the experimental setup.
Decision: Accept (poster)