Faster Training of Word Embeddings
Authors: Eliza Wszola
Martin Jaggi
Markus Püschel
Venue:  (2021)
Keywords: multicore
performance
machine learning
word embeddings
word2vec
fasttext

Abstract: Word embeddings have gained increasing popularity in the recent years due to the Word2vec library and its extension fastText that uses subword information. In this paper, we aim at improving the execution speed of fastText training on homogeneous multi- and manycore CPUs while maintaining accuracy. We present a novel open-source implementation that flexibly incorporates various algorithmic variants including negative sample sharing, batched updates, and a byte-pair encoding-based alternative for subword units. We build these novel variants over a fastText implementation that we carefully optimized for the architecture, memory hierarchy, and parallelism of current manycore CPUs. Our experiments on three languages demonstrate 3-20x speed-up in training time at competitive semantic and syntactic accuracy.

Under review as a conference paper at ICLR 2021
FASTER TRAINING OF WORD EMBEDDINGS
Anonymous authors
Paper under double-blind review
ABSTRACT
Word embeddings have gained increasing popularity in the recent years due the Word2vec library and its extension fastText that uses subword information. In this paper, we aim improving the execution speed of fastText training on ho- mogeneous multi- and manycore CPUs while maintaining accuracy. We present a novel open-source implementation that flexibly incorporates various algorith- mic variants including negative sample sharing, batched updates, and byte-pair encoding-base alternative for subword units. We build these novel variants over a fastText implementation that carefully optimized for the architecture, memory hierarchy, and parallelism of current manycore CPUs. Our experiments on three languages demonstrate 3-20x speed-up in training time competitive semantic and syntactic accuracy.
INTRODUCTION
Word embeddings have a long history (Rumelhart et al. 1986 Bengio et al., 2003 Collobert & Weston 2008), but have received much attention in recent years due word2vec (Mikolov al. 2013) and its computationally efficient implementation via skip-gram with negative sampling. Word embeddings capture contextual relationships between the words, and have become standard input representation for the majority of NLP tasks, benefitting, e.g., classification (Joulin et al., 2016 Deriu et al. 2017) machine translation (Jansen] 2017, Conneau al. 2017). More recently, state-of-thc-ar results on many language understanding tasks were achieved by deep transformer architectures such as BERT (Devlin al., (201)). which however are very compute intensive both at training and inference time, even with pre-trained models and reduced parameter space. Thus, simpler and more lightweight static word embeddings such as fastText (Bojanowski et al., 2017) are still widely used, due to their fast execution, comparable results for particular tasks (Tseng et al.) 2019), and ability to produce a single vector per word, which helps in information retrieval with interpretabilityy and search index construction.
Contributions. In this paper, we present algorithmic and code optimization techniques to improve the training time of word2vec and fastText embeddings modern general-purpost multicore and manycore computers. We present an optimized open-source implementation word2vec and fast- Text that encapsulates number of algorithmic variants including negative sample sharing, batched updates, and subword units based on byte-pair encoding approach. Our extensive evaluation on three languages shows that the best combinations of optimizations speed up training time by 2.7- 20.6 times while maintaining accuracy of selected NLP tasks.
2 WORD EMBEDDINGS
Word2vec. Word2vec is built upon a simple bilinear regression model trained on word co- occurrence, resulting in numerical feature representations as floating point vectors of dimension- ality d. Given word in sentence, the goal of the algorithm is to maximize the likelihood of predicting surrounding (context) words. To achieve this, the model is trained to increase the prob- ability of predicting particular words if they appear close corpus. A popular variant also decreases the probability of predicting words that do not appear close to the current word (negative sampling (Mikolov et al.) 2013 Goldberg & Levy 2014)).
given current word in the training
During training. the algorithm processes the corpus in streaming fashion. Each
word w; (called current word) is processed together with its surrounding context words

---

Under review as a conference paper at ICLR 2021
a²'
"a'
jumps'' quick"'
jumps'
quick
'quick'
Min
'dog'
'dog'
Mout
brown
'brown'
'brown'
fox
'fox'
'fox'
jumps
...
hash("< <<ro'")
Figure 1: Representation of source and target words in the input (Min) and output (Mout) matrix in fastText (skip-gram). The sentence is "a quick brown fox jumps over the lazy dog", the current word w, is "brown" and the context window size is 2. The words in the corpus are represented as indices of the corresponding rows in Min and M
[Ws-Cs... w;-1}, (wi-1,-.. w++c), where C' is the range of the context window. There are two modes of operation training the model for the following prediction tasks:
Skip-gram (SG): predict target context words using the current word wy as the source. CBOW: predict the target current word wi; using context words the source.
Each word w in the vocabulary of size V is represented as source w, by one row the V x input matrix Min containing word embeddings, and each word represented row in the V x d output matrix Mout that is used to calculate the training objective function. The goal of the optimization
target wa by one
to maximize the inner products (minimize difference) of real pairs of source current words with the target context words or vice versa, using the binary logistic loss. This approach can be improved by the use of negative sampling, where the algorithm additionally maximizes the difference between the source current words and the words picked randomly out of the source's context. Training performed using stochastic gradient descent (SGD). SGD is performed in parallel with threads by splitting the training corpus into parts and processing them asynchronously ("Hogwild" approach (Recht al, 201T)). The final embedding of each word its corresponding row in Mi. Moat is discarded at the end of training.
Text.ext. FastText (Bojanowski et al. 2017) improves word2vec by utilizing subwords of target words during the training. A typical run of fastText uses subwords of lengths k = 3 .... 6, containing delimiters (.) at the boundaries. For example, for the word paris and = 3 the subwords are: (pa, par, ari, ris, is).
In fext, the embeddings Mn, are extended to contain rows representing both entire words as well hashes of all their subwords. Additionally, the representation of the entire word is added the set of its subwords. During the execution of the algorithm. the hidden layer h is built by averaging vectors in M,, representing the source word's subwords. The final vector embedding for each word is obtained in the same way. Mout remains unchanged. Fig. I shows an example of how the word vectors are stored and accessed. single update described in Alg.
Related work. FastText has been implemented as part of the popular Gensim library (Rehurek Sojka| 2011) using Cython and default machine's 's BLAS library (e.g., Intel MKL) for algebraic operations. In our experiments we found the code memory- expenssee and slow: training 5 epochs on a GB English Wikipedia dump with 24 threads took approximately
hours on Knights Landing CPU, about 10 times slower than the original fastText. Therefore, we use the original code provided by Facebook Research (20 as the baseline in all our experiments.
For skip-gram gram with negative sampling, PWord2Vec (Ji et al.) 2016) transforms the "Hogwild" ap- proach into "Hogbatch", , by performing updates on multiple context words at once (effectively turn- ing series of dot products into matrix-matrix operation) and sharing negative samples for the entire batch. We employ similar techniques in our implementation.
The work by Rengasamy et al. (2017) extends this approach by context combining, where multiple contexts can share a set of negative samples and be updated all at once. We do not adapt this approach it requires careful preprocessing rewarded by relatively small speedup over PWord2
2

---

Under review as a conference paper at ICLR 2021
Algorithm 1: A single iteration of the original fastText algorithm. In skip-gram, it is performed on each current-context word pair (as source-target). In CBOW, all context words are used as source words the same time.
Data: source word(s) W,, larget word wa, learning rate I, number of negative samples w if skip-grum then
// Initialize.
h Ma(w,) ( S (ur,) Ma(a)
// Average vectors of the source word(s) and their
count coum! (uuuuunsss
else CBOW then
// subwords obtain the hidden layer.
E, (Malma)+ Cumnndddtt Ma(s))
La (countt sabwonds (wx
0
// Reset the gradient.
// Update the target word.
1(1 o(h- Mon(we)))
// Compute positive seore reflecting similarity between
// and the row Mow( (we) representing wt.
g- = Muu((1)) (wx,) ) Mon (wx,) + a
// Build the gradient.
// Update the target word.
for + to n do
I/ Update negative samples: negative score.
10
pick a random negative sample wy
11
1(0 (h Moul (we/ ()) (2x)))
I/ Compute negative score.
g =
// Build the gradient. // Update target word.
13
Moar (we+) Mons (wx) t..
14
end
15
skip-gramm then
// Update the source rows(s).
16
(w.) Mi (w.)). +
17
foreach = E subwords (wa) do
18
(= = Min + t
end
20
else if CBOW then foreach wi do
21
// As a result, the difference between rows in Mn(ws) // and Mou which corresponds to the positive samples // drops, and the difference between rows in Mas // which correspond to the negative samples increases.
(w(w:)) (w:,) +g
23
foreach 2 subwords(uu w.)) do
24
(=) = Mn(2) + g
25
end
end 26
Word2vec and fast'Text have been also implemented for GPU clusters. BlazingTexxt (Gupta & Khare, 2017) tackles the problem of efficient batch size and synchronization for multiple GPUs. While this issue is of no concern on CPU, they report the execution time on single GPU comparable to 16- threaded CPU fastText baseline. We further speed up the CPU implementation. The work by Bac & (2166) reports up to 11x speedup of word2vec with negative sampling single-threaded CPU word2vec. However, they report only up to 1.6x speedup over a 12-threaded CPU run. Our subword version of the code roughly 5 (skip-gram) and (CBOW) faster than the 20-threaded runs of the original word2vec. Word2vec and fastText are memory-intensive algorithms. Additionally, fine-grained parallelism is limited by relatively small vectors typically used in the computations. These characteristies severely limit the potential advantages of GPU over CPU.
K20 GPU over the
Li al. (2019) discuss distributed version for many GPUs aiming at the reduction of write conflicts updates. Similarly (and independently), we made attempts pre-scheduling list of current- context word updates, but we found the overhead of this preprocessing prohibitive. Nonetheless, the algorithmic variants presented our paper can be applied distributed setting, as long as the data used for single update fits inside batch used in the distributed computation. This will be the case for our variants, since they either execute separate updates on each current-context word pair, or update current words with their entire (typically small) contexts. This is also the case in the original fastText and therefore, the communication cost should not increase.
Another popular word embedding model is GloVe (Pennington et al. 2014). While the Authors claim superiority over word2vec, a more thorough evaluation (e. g.. by Wang et al. (2019), or Kumar et al. (2020)) shows that there is clear winner, as the results may vary depending on the training corpus, evaluation task and hyperparameters used. Additionally, Glo Ve lacks the information on word morphology provided by fast/exxt, and does scale well for large vocabularies. Since GloVe
3

---

Under review as a conference paper at ICLR 2021
original
original
code_opl
code_opt
no no_subword
NS .
DH
BPE.
no.subword
batch
BPE_
DH.NS.A
DHF
NS.CT
DHIFNN DII..S.s
(a) skip-gram (SG)
(b) CBOW
Figure 2:
Dependency between our code variants of skip-gram and CBOW. The experiments for the "NS" variants (no box frames) are only shown in the appendix due to inferior experimental accuracy.
based on a completely different algorithmic structure, that is. creation and reduction of global word co-occurrence matrix, there is no straightforward way to apply to it our code optimizations and variants.
3 OPTIMIZATION TECHNIQUES AND ALGORITHMIC VARIANTS
To improve the training time, we first identify the most expensive operations. Assume that the source wsrd(s) have total of m subwords and that we use n negative samples per target word. Then each update comprises:
Construction of h: sum of m vectors (line 2 or
Loss calculation: n + dot products and 2 + vector additions (lines 6
13
Gradient update: m vector additions (lines 16 or 26). 26).
In skip-gram, h is built only from a single current word, while in CBOW, it is constructed from all context words. The loss function, in contrast, is computed once per each current-context word pair in skip-gram, while in CBOW, the loss is computed using the entire context. This means that for CBOW, the construction of h and the gradient update consumes most of the execution time, while for skip-gram, these operations take roughly the same amount of time the loss function calcula- tion (assuming the default parameter of = 5). operations listed above are memory-intensive and therefore memory bound: thus, the best approach to optimize them is by reducing memory movement and avoiding unnecessary updates. is further supported by our observations during the tests isolated sections of the code. We noted that large amount of execution time is taken by the latency of lower levels of memory hierarchy when accessing data from the rows scattered across Min and Mour, which is access pattern required provide high quality embeddings.
To speed up the training, we first perform a number of code performance optimizations and then various algorithmic modifications compared the original fastText. Some modifications depend on each other as illustrated in Fig. _ Some, but not all, techniques apply to both modes of opera- tion. All our improvements build on code_opt which a CPU-specific optimization of the original fastText code. For skip-gram, we consider a batch variant and negative sharing across the context (NS.CT) For CBOW, we consider keeping track of the values in the hidden layer and updating them dynamically rather than building this layer from scratch in each iteration (DH: variable con- text window size, DHF: fixed context window size). We consider combinations of this technique with negative sharing involving different number of positive samples that the negative samples are shared between (NS.s). Additionally, for both CBOW and skip-gram, we test no oo_subword where remove the subwords from code_opt, making equivalent to optimized word2vec, and BPE,h, where we replace samples with BPE tokens obtained from a pre-trained token set of size
We next discuss these variants, referring specific parts of Alg. T that they modify.
Code performance optimizations (code (code_opt). For efficient execution we explicitly vectorize ma- trix and vector operations using 512 intrinsics. We block and merge operations involving mul- tiple reads from the same location in memory to make them more cache- friendf (temporal locality), like averaging the rows of Mn, subsequently reading from and writing to Maut-
4

---

Under review as a conference paper at ICLR 2021
During the creation of hidden layer h (line 2 or 4. each element of h is stored only once during summing up the vector representations of subwords. This improves the original code which performs a separate store for each subword.
reduce the number of array accesses such that
To speed up the binary logistic loss function, we vectorize the dot product (lines 6 with the use of eight accumulators to increase instruction-level parallelism without too much register pressure.
We merge the update of the gradient g and the relevant rows of Mour (lines 7 12
to avoid
multiple reads from the latter. We still call the loss function once per each wy, and w/-
Similar to the creation of h, we improve the update of Min (ws.) with g (line 16 or 21 22 each element of g only once for all words and subwords W/s.
by reading
The optimizations in the version code_opt are used in all algorithmic variants discussed next. Note that these optimizations can also be applied other regularization schemes such as the hierarchical softmax used in the original word2vec (Mikolov et al. 2013).
No subwords (no_subword)) Experiments in Section show that it is sometimes useful to train word embeddings without any subword information. We provide a code variant which disables sub- words, but applies all optimizations discussed above. It algorithmically equivalent to word2vec. In Alg. 1 subwords (w.) in lines and becomees an empty set and needs not be processed. While this is expected to improve the training time, especially for CBOw which dedicated a large part of its runtime to averaging and updating subword representations, the information on the word morphology becomes scarce. We will later see that the word embeddings trained without subword information do not perform well when used for syntactic tasks. On the other hand, the training then focuses on semantic information which is reflected in higher semantic quality of these embeddings.
Minibatching (batch for SG). For skip-gram, we implement a form of minibatching of the target words per each source word. Rather than following the work of Ji et al. (211)). which merges all (n.w)) ) rows in minibatch into matrix, we follow the original fast Text's approach hitherto only applied to CBOW, which simply averages all these rows. The advantage of our minibatching over the original fastText skip-gram being able to execute single update for each context window of the current word w,, rather than per each current-context word pair. This means that h and the relevant rows of the input matrix Mnn are updated only once per each current word, independent of context window size. Lines 2 and 16 are now executed only once per context window, in similar fashion as in lines 4 and 21 22 respectively). This creates an additional delay between reading and writing word's i subword representations increasing the possibility of write conflicts, but our experiments later show that the accuracy remains nearly unaffected. Minibatching can bring significant speed improvements to subword-based training due to the relatively high cost of building h and updating all subword representations. As mentioned, in fastText CBOW, this form of batching is already a part of its algorithmic structure.
Negative sharing CNS.TT and NS_s). We implement negative sharing proposed by Ji et al. (2016), but adapted for and built over SG_batch and the natural batching of fastText CBOW. For skip -ragr,mm, we share negative samples among all words in the entire context window of w; (NS CT). For CBOW, share negative samples for consecutive current words wi (NS 8). In our implementation, is hyperparameter chosen by the user. Thus, line 10 executed only n times every s-th update. While negative sharing results in fewer expensive random memory reads and improved memory locality (c. g., for d = 300, n = 5, and a context window size of 11, the data worked upon takes up c. a. 16 KB, while an usual L1 cache size is not less than 32 KB), it proportionally reduces the number of data samples used in training per current-context word pair. For this reason, despite improvements in execution time, NS yields inferior accuracy. Therefore, we do not report its results the paper, but only present them in the appendix.
Dynamic hidden layer update (DH). CBOW spends large portion of its execution time building h and updating relevant rows of Min for each subsequent current word w; and its context window. Therefore, we opt for adding and removing subwords only as their words move in and out of the context window as the algorithm processes the training text. After each shift of the context window, we update the rows of Mn for all removed subwords, readjust to the gradient g, and add new subwords to h. Thus, rather than performing the entire sum in line 4 the data is processed in five steps. Assuming that x embeddings remain inside the context window after a particular shift:
1. Denormalize h.
5

---

Under review as a conference paper at ICLR 2021
2. Update Mn for subwords falling out of the context window.
3. Subtract embeddings of subwords falling out of the context window from h.
4. Readjust to gradient g: h = h + zg.
5. Add subwords falling into the context window to h.
6. Normalize h.
Note that this creates additional delay between reading and writing to the rows of Min but empiri- cally it does not harm the vector quality.
Fixed window for dynamic hidden layer update (DHF), Since the window size is picked randomly in each iteration, some words will fall in and out of the context window multiple times, forcing DH to remove and add the same subwords to h multiple times over a short period of time. To mitigate this, we fix the window size. While potentially saving time, this approach comes with a pitfall: the variable window size is a natural way of sampling context words that are closer to current word w; with greater probability, which reflects a greater contribution of these words to the current word's meaning. DHF effectively ignores the impact of the distance of context words.
Byte-Pair vocabulary (BPE h). We also propose an alternative approach to subword embeddings. replacing the subwords by Byte-Pair Encoding (BPE) tokens (Sennrich et al. 2016). These are produced with the Hugging Face Tokenizers library (Moi) 2019) in the form of token IDs for the most frequent word fragments, where h is hyperparameter. We expect this to reduce execution time and memory consumption as the number of tokens is typically an order of magnitude smaller than that of subwords. To our knowledge, this is the first attempt to apply BPE tokenization provide additional subword information in fastText-like fashion. In our experiments in Section 4 we train the tokenizer over the same training corpus as our embeddings, but both trainings could use different corpora. In case the BPE variant of fastText is unable to tokenize a word found in its training corpus (e.g., because it was absent from the corpus used for training the tokenizer). the word remains as it is. without additional embeddings. An alternative approach would be to create embeddings for tokens consisting of single characters: however. we found that if many words fail to be tokenized, this may cause a drastic slowdown, likely due to update conflicts on the single-character tokens. In Alg. I using the BPE variant means replacing "subwords" with "tokens".
4 EVALUATION
In this section we evaluate our performance--ptimnzze polyalgorithmic implementation for training word embeddings current homogeneous multicore system. For each algorithmic variant, and considering three languages, we report the speedup we achieve for training and the obtained accuracy of the generated word embeddings w.r.t. number of semantic and syntactic quality tests.
Setup. We use a dual-socket Intel(R) Xeon(R) Silver 4114 CPU processor (Skylake, 20 physical cores). For evaluation, we create an English corpus described by Facebook Research (2016b). For other languages, we proceed in analogous fashion: download respective Wikipedia dumps (Wikime- dia Foundation 2001), sanitize and lowercase with the script wikifil.pl authored by MMahnnyy| (2006).
For each language. the script is modified to capture relevant characters and replace relevant words. We truncate the outputs words for English, (b) 592,674 words for German, (c) 385,596 words for Russian. The purpose of our experiments is to speed up training over the original fastTexx. We demonstrate the speed-ups of our implementation and show which algorithmic variants maintain accuracy at the same time.
billion characters. The resulting vocabulary sizes are: (a) 11
The results for English are presented in Table The names of algorithmic variants match those from Section 3 and Fig. 2 We omit negative sharing (NS) due to low accuracy. but show the results in Appendix
F In tokenized runs (BPE), we use / = 20K, 200K. All other hyperparameters are the fastText defaults. The speedups shown over the original implementations SGroriginal and CBOW _originnal, respectively, from Bojanowski et al. (2017). run with the same number of threads. The scaling column shows the speedup of our code when run with 20 threads compared to I thread. The runtimes are consistent over several runs and are shown in Appendix F
We perform various semantic and syntactic accuracy tests explained below. The best accuracy scores for each test are marked in blue. Pareto-optimal combinations of accuracy scores are shown bold- faced. Pareto-optimal means that no other algorithmic variant dominates it, i.e., is better on each

---

Under review as a conference paper at ICLR 2021
Table I: Accuracy and speedup achieved with our library over fastText when training on English Wikipedia corpus. Blue: best accuracy in category, bold: Pareto-optimal accuracy, speedup is over the original fastText run with the same number of threads, and scaling is the speedup of 20 threads vs. thread for our code. Higher is better for all metrics.
algorithmic
accuracy
speedup (times)
scaling
variant
QW
BATS
20
MUSE
Battig
sem.
syn.
syn.
thread
threads
SGooriginal
27.07
64.22
9.30
41.54
0.643
40.97
1.0
1.0
15.2
Our work:
26.37
63.75
9.29
41.61
0.647
40.85
3.7
2.7
11.2
SG no www.noouuubae subword
47.42
48.07
12.97
27.81
0.635
41.27
10.2
8.8
13.1
(G_hbtch
24.28
62.47
8.92
41.84
0.630
40.76
4.4
3.6
12.2
SG 1G_BPE.20K
40.78
53.85
12.86
34.50
0.642
40.43
4.9
4.2
13.0
SG B2BBPP.4K
47.77
50.56
13.33
33.16
0.646
39.80
4.8
4.2
13.2
SG.BPE.20OK
54.98
44.79
12.65
27.73
0.634
41.46
4.8
4.2
13.4
CBOW original
9.27
67.74
5.53
63.42
0.546
32.98
1.0
13.2
Our work:
CBOW _code_ppt
9.55
68.14
5.69
63.52
0,538
33.07
4.6
2.3
6.6
CBOW _no_suwwrrd
51.00
54.49
14.84
32.29
0.614
40.68
20.6
20.4
13.1
CBOW DD
11.38
68.73
6.51
63.58
0.583
36.30
4.8
2.7
7.4
CBOW DHF
5.29
57.44
3.94
51.71
0.633
31.37
3.3
9.2
CBOW BPE 20K
22.65
46.01
10.17
36.98
0.607
35.56
11.4
11.0
12.8
CBOW BPE 40K
30.85
40.47
10.13
33.38
0.614
36.07
11.5
11.1
12.8
CBOW BBP.
43.33
26.40
8.12
24.70
0.606
40.39
11.4
11.3
13.0
score. Together with Fig. 2 the tables also show the incremental impact on accuracy and execution speed of each variant.
Accuracy tests. We perform multiple evaluation tasks to test the quality of our embeddings. First, test embeddings with the word analogy task script provided with word2vec (Mikolov al. 2013) for both semantic and syntactic accuracy. For English, we use the questions words (QW) dataset (Mikolov) 2013). For other languages, we use its translations: German (Köper et al. 2015)
and Russian (Kononova 2017). Additionally, we employ the Vecto library |Vecto][2183 to evaluate on the The Bigger Analogy Test Set (BATS) (Gladkova et al., 2016) on English embeddings, with the 3CosAdd method. For all analogy benchmarks, we observe that fastText performs better for syntactic than semantic tasks as was already noted by BBjjanwwskii et al. (2017). Second, we compute word similarity scores with Facebook MUSE (Conneau et al.) 2017), using monolingual evaluation with word similarity tasks on semantic datasets. For English and German, we use the tests provided by MUSE. For Russian, present the averaged MUSE output. For detailed score for each MUSE test set, see Appendix B Third, we the scripts provided by the Word Embedding Benchmarks package (Jastrzebski 2015) to perform the concept categorization (word clustering) task. We evaluate on the semantic Battig test set introduced by Battig Montague (1969).
the HJ dataset (Panchenko et al.) 2016). The tables in this section
Evaluating English skip-gram. First, we evaluate multiple variants of skip-gram presented in the first section of Tab. [ The dependencies between the variants illustrated in Fig. 2(a) We observe that only optimizing for efficient execution (code_opt) yields for fast 2.7-3.7 7 speedup while maintaining accuracy. The no_subword variant yields 8-10x speedup over original and about 3x speedup over code opt. For word analogy, no subword improves the embeddings semantically. The tokenized versions roughly balance between the fastTexx-- and word2vec-style embedding quality, with an exception of BPE_200K where the number of tokens is close to the vocabulary size, ef- fectively turning only the most common subwords into separate tokens. This approach provides a semantic accuracy even greater than original for both BATS and QW, however at price of syntactic quality, all including roughly 4-5x speedup over original and up to 1.5x speedup over code cot._opt. For QW, the accuracies vary greatly, while BATS indicates that smaller number of tokens is gen- erally preferable. The batch variant maintains or slightly handicaps the accuracy of fast and provides a slightly smaller speedup than the tokenized versions. The different variants of skip-gram

---

Under review as a conference paper at ICLR 2021
perform almost equally well on the word similarity and categorization tasks, and all of them yield Pareto-optimal results. All variants show good parallel scaling.
Evaluating English CBOW. The CBOW results shown in the second section of Tab. D the dependencies between variants are in Fig. 2 The code opt variant yiclds 2.3-4.6x speedup over original, less than for skip-gram, but the obtained accuracy not Pareto-optimal. For word analogy, CBOW generally performs better on syntactic than semantic questions. The no subwwrrd variant provides good scaling, and over 20 speedup over original and about 8-9x speedup over code code_opt. It diminishes the discrepancy between these scores, albeit impacting negatively the syntactic quality of the embeddings, while achieving highest scores for word similarity and categorization. None of the tokenized variants was able to beat no subword both in speed and evaluation on these tasks, but they provide an improvement in semantic accuracy over original, as well as in word similarity and categorization. The BPE variants achieve roughly 11 x speedup over original. The DH variant provides only a slight speedup over code_opt (2.7-4.8x speedup over original), but yields higher accuracies in all tasks, while DHF impacts negatively all scores except for MUSE, but provides speedup over DH with multiple threads.
Comparison between skip-gram and CBOW. As rule of thumb, the fastText implementations of skip-gram perform much better on semantic questions in word analogy tasks and slightly bet- ter in word similarity and categorization tasks. For syntactic questions, the CBOW code opt and CBOW DH variants are a better option. On the other hand, CBOW no subword performs nearly well for word similarity and categorization tasks skip-gram. Therefore, in specific cases, the former can be used in licu of skip-gram to boost the execution speed.
results for German and Russian, presented analogously to those in English. In terms of evaluation accuracy, they are largely con- sistent with English, with the small exception of CBOW BPE_20K, which performs better than so_owoosubwoor on the German corpus. This indicates the impact of the number of tokens used during training and opens opportunities for further investigation. Noteworthy, CBOW DH achieves the best scores on syntactic tasks for all evaluated languages. Using -kip-gram with BPE tokens rather than fastText-style subwords performs very well in terms of both speedups and aceu- racy scores, all of which are Pareto-optimal. The code_opt variants yields slightly better speedups than for English, and further optimizations lead to significantly greater speedups. The code_opt variants yield roughly 3.5-5x speedup their respective original versions. The best achieved improvement is CBOW_no _no_subword, up to 50x for Russian. This shows that our improvements particularly beneficial for morphologically-ric languages with large number of subwords per word.
Evaluation on German and Russian corpora. Table
5 CONCLUSIONS
We presented a thorough evaluation, and associated open-source implementation, of various op- timization techniques for fastText and word2vec. particular, these include code-level perfor- mance optimizations and the use of BPE tokens rather than subwords. For example, for English, our code offers practitioners speedups in the range 2.7-20.6x, while maintaining a single or multi- dimensional notion of accuracy. We achieve good parallel scaling, which is expected to bring even more benefits in the future, as the number of cores further increases. The choice of algorithm de- pends heavily on the accuracy metrie: for all languages, there is no universally best variant, which makes case for our polyalgorithmic implementation and thorough evaluation of trade-offs. Our techniques should also apply sent2vec (Pagliardini et al. 2018) for sentence embeddings.
REFERENCES
Maria Antoniak and David Mimno. Evaluating the stability of embedding-based word similarities. Transactions of the Association for Computational Linguistics, 6:107-119, 2018. 2018.
Seulki Bae and Youngmin Yi. Acceleration of word2vec using gpus. International Conference Neural Information Processing, pp. 269-279. Springer, 2016.
8

---

Under review as a conference paper at ICLR 2021
Table Accuracy and speedup achieved with our library over fastText when training on German and (b) Russian Wikipedia corpora. Blue: best accuracy in category, bold: Pareto-optimal accuracy, "speedup": over the original fastText run with the same number of threads, "scaling": speedup 20 threads vs. I thread for our code. Higher is better for all metrics.
algorithmic
accuracy
speedup (times)
scaling
variant
QW
MUSE
syn.
thread
threads
SG_original
21.13
49.94
0.587
1.0
1.0
8.5
Our work:
KG_codc_opt
19.41
49.12
0,575
3.5
4.9
11.8
SG [G__no_sbbwon
42.31
27.29
0,589
11.5
18.8
13.9
bG.batch
17.43
49.13
0.573
4.6
7.0
12.9
SG_BPE Br_BPE.20K
29.91
36.83
0.589
5.5
8.9
13.9
SG 4P..P
34.55
29.96
0.593
5.2
9.0
SG_BPE.200K
45.90
23.39
0,593
5.3
9.0
14.6
CBOW original
4.71
58.94
0.507
1.0
1.0
8.3
Our work:
CBOW -code_ode_opt
4.30
59.09
0.510
4.3
3.4
6.7
CBOW_no_subwe _no_subword
37.75
27.66
0.559
24.1
41.8
14.3
CBOW_D __D
5.96
59.81
0.527
4.9
4.2
7.0
CBOW DHF
1.86
54.17
0.591
5.2
4.5
7.2
CBOW BPE. 20K
11.19
31.31
0.542
13.1
21.1
13.3
CBOW BPE 40K
18.67
23.25
0.557
13.3
22.0
13.7
CBOW BPEK
27.81
16.14
0.560
12.9
22.6
14.5
(a) German
algorithmic
accuracy
speedup (times)
scaling
variant
QW
20
MUSE
sem.
syn.
thread
threads
SG-original
12.29
77.61
0.633
1.0
1.0
10.2
Our work:
SG_code_opt
12.86
77.81
0.622
4.1
4.2
10.4
SG_no_subword
23.82
42.89
0.588
12.1
14.0
SG batch
10.58
76.32
0.629
5.1
5.2
10.5
SG BPE 20K
14.22
51.86
0.621
5.8
7.9
13.8
SG_BPE_40K
16.75
46.81
0.608
5.8
7.9
13.9
SG_BPE_200K
22.90
45.91
0.600
5.8
8.0
CBOW original
8.88
80.78
0.495
1.0
1.0
7.9
Our work:
CBOW CBOW_code.opt
9.18
79.79
0.497
4.5
4.6
8.0
CBOW_m no.ssuborrd
16.01
37.35
0.532
28.1
50.5
14.2
CBOW_D DH
9.17
81.43
0.523
4.3
5.7
10.5
CBOW DHF
7.32
78.75
0.557
4.6
6.2
10.6
CBOW BPE 20K
8.28
34.82
0.513
15.0
25.2
CBOW BPE_40K
8.96
40.66
0.493
15.2
26.1
13.6
CBOW BPE. 200K
11.13
35.96
0.537
15.3
27.0
14.0
(b) Russian
William F Battig and William Montague. Category norms of verbal items in 56 categories replication and extension of the connecticut category norms. Journal of experimental Psychology. 80(3p2):1, 1969.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb): 137-1155, 2003.

---

Under review as a conference paper at ICLR 2021
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computatiooal Linguistics. 5:135 146. 2017. doi: 10.1162/tacl 00051. URL https: /www www.wblwwwbbcc.... wwwp:eb...gg/anthhhoooeoo
Q17-1610
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning. pp. 160-167, New York, NY, USA, 2008 Association for Computing Machin- cry. doi: 10.1145/1390156 139011777
Alexis Conneau, Guillaume Lample, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. arXiv preprint arXiv:1710.0408/ 2017.
Jan Deriu, Aurelien Lucchi, Valeria De Luca, Aliaksei Severyn, Simon Müller, Mark Cieliebak, Thomas Hofmann, and Martin Jaggi. Leveraging large amounts of weakly supervised data for multi-language sentiment classification. In Proceedings of the 26th International Conference on World Wide Web, pp. 1045-1052, Republic and Canton of Geneva, CHE, 2017. Interna- tional World Wide Web Conferences Steering Committee. doi: 10.1145/3338112.30555 URL htttp:://bi. www.:o.gggoooo
3052611
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. pp. 4171-4186, 2019. doi: 10.18653/v1/ N19-1423.
Facebook Research. fucccboookkeeeeurc//fffStettt Library fast text represemtation and classifica- tion., 2016a.
Facebook Research. Word representations, 2016b.
batps///aattext..ccoo/////
mnsupervised-tutorial html
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy-based detection of morphologi- cal and semantic relations with word embeddings: what works and what doesn't. In Proceedings of the NAACI. Student Research Workshop, pp. 2016.
Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al.'s negative-sampling word-embedding method. urXiv preprini arXiv:1402.3722. 2014.
Saurabh Gupta and Vineet Khare. Blazingtext: Scaling and accelerating word2vec using multiple gpus. In Proceedings of the Machine Learning on HPC Environments, pp. 1-5, 2017.
Stefan Jansen. Word and phrase translation with word2vec. arXiv preprint 17x1v:1005.03127 2017.
Stanislaw Jastrzebski. uating word embeddings. 2015.
Package for eval- https: htttps://github.coo ko.co///kukuakk///
word-embeddings-bae wew.hnggn-bbbcmmarr......
Shihao Ji, Nadathur Satish, Sheng Li, and Pradeep Dubey. Parallelizing word2vec in multi-core and many-core architectures. arXiv preprint arXiv: 16X1:v66111.6172 1.06172, 2016.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.ip: Compressing text classification models. arXiv preprint 1rXiv:661..03651, 2016.
Taryana Kononova. Russian adaptation of Google Analogies Dataset, 2017. Fusvectores . wttp://taatteeesseeeetoto saceetssruuuaaae ogy oog_ooggy__t_aaaaeeeeeeeeee PROPN. txt
https://
Maximilian Köper, Christian Scheible, and Sabine Schulte im Walde. Multilingual reliability and
"semantic" structure of continuous word spaces. In Proceedings of the I 11th international confer-
ence on computational semantics, pp. 40-45, London, UK. 2015. Association for Computational acc.alllall weeb.webb... www./aggggggttoooooooeo hotogy wwl.ooyyyyllliie 0105
Linguistics. URL ht
ant
P Santosh Kumar, Rakesh Bahadur Yadav, and Sunita Vikrant Dhavale. A comparison of pre-trained word embeddings for sentiment analysis using deep learning. In International Conference on Innovative Computing and Communications, pp. 525-537. Springer, 2020.
10

---

Under review as a conference paper at ICLR 2021
Bofang Li, Aleksandr Drozd, Yuhe Guo, Tao Liu, Satoshi Matsuoka, and Xiaoyong Du Scaling word2vec on big corpus. Data Science and Engineering, 11) 2019.
Matt Mahoney, About the Test Data, 2006. httppss//mtttaabbrrreeee//////ee
html Happendixa
Tomas
Mikolov.
Tuestions-Words.7X1 2013.
https://githun com/
nicholas http://swooouute///oocee///////a
/wtpeeerrrreeeeiinnnnoo wornns-worddstte txt
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen-
tations in vector space. arXiv preprint arXiv:1301.3781, 2013.
Anthony Moi. Fast State-of-the-Art Tokenizers optimized for Research and Production, 2019. zers
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. Unsupervised learning of sentence embed- dings using compositional n-gram features. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long Pupers), pp. 528-540, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v1/118-1D0 1049. URL htLhttps:// www.e..
org/ Bogg/anhhooogy/NB---B1
Alexander Panchenko, Dmitry Ustalov, Nikolay Arefyev, Denis Paperno, Natalia Konstantinova, Natalia Loukachevitch, and Chris Biemann. Human and machine judgements for russian semantic relatedness. In International conference on analysis of images. social networks and texts, pp. 221 235. Springer, Cham, 2016. Utthttps: doi or0.10..r11..000 1007
21
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1544 1543, 2014.
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. Advances in Neural Information Processing Systems 24, 693-701. Curran Associates, Inc., 2011. htppt U
pdf
Radim Rehurek and Petr Sojka. Gensim-statistica semanties in python. Retrieved from genism. org, 2011.
Vasudevan Rengasamy, Tao- Yang Fu, Wang-Chien Lee, and Kamesh Madduri. Optimizing word2vec performance on multicore systems. In Proceedings of the Seventh Workshop on Irregu- lar Applications: Architectures and Algorithms, pp. 1-9, New York, NY, USA, 2017. Association for Computing Machinery, doi: 10. 10.1445/3449044.344977 URL /tLptts:///6ii..////1
1145/3149704.1 31497666.114
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. Nature, 323(6088) 1986. htt.://tss///cii.or//ooo...
B23533a0
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. pp. 1715-1725, 2016. doi: 10. 12.186/3///116-11622 URL httph:tttt/w/tt
acl web. www.//anttiitt www.nthooooyyPPee oog.oooyyyo11/o
Hou-Chiang Tseng, Hsueh- ChuchhCChh Chen, Kuo-En Chang, Yao-Ting Sung, and Berlin Chen. An in- novative bert-based readability model. In International Conference Innovative Technologies and Learning, pp. 301-308. Springer, Cham, 2019. URI. httpp::/ //oii /ttps/////cc..oo11100000000
978 378-3-300-354333 32
Vecto. vecto-ai/vecto: Doing things with embeddings., 2018. https://github. com/ vecto-ai/vecto.
11

---

Under review as a conference paper at ICLR 2021
Bin Wang, Angela Wang, Fenxiao Chen, Yuncheng Wang, and C-C Jay Kuo. Evaluating word embedding models: Methods and experimental results. APSIPA transactions on signal and infor- mation processing. 8. 2019.
Wikimedia Foundation. Wikimedia Downloads, 2001. https: wwk.media..moo
A DETAILS OF EXPERIMENTAL SETUP
For the experiments with questions-worts and MUSE, we do not remove any questions containing vut-ol-vocabulary words. We use full test data sets for modes, therefore our results are consistent.
For MUSE, we use the following tests provided by the library
English:
EN. YP-130
EN_ WWSSIMLEX-999
EN-Muurk-771
EN RG-65
EN VERB-147
EN ENSEEMEAL117
EN MTurk-287
EN MC-33
EN. EN_RW-STAN EW-SSTNFORR
EN_WS- NT.WS33--SII.
EN-TR-3K
EN WS-353-ALL
EN. WS-353-REL.
German:
DE ZG222
DE GUR65
DE_SIMIEXX-99
DE. DE_GUR350
DE.SEMEVALI7
DE WS-353
For the final MUSE scores, we take arithmetic mean of individual scores obtained from these tests. For Russian, MUSE provides tests. Therefore, download and the HJ datase 2
B DETAILED EVALUATION WITH MUSE
Tables and present detailed MUSE results for each test set. For Russian, set, thus the results would be redundant with those Section 4 and Appendix [F] For the English and German tables, therefore, the individual scores are not expected to average exactly to those in Section 4
only test
the experiments embeddings obtained in different training run:
C PREPARATION OF DATA
To prepare German and Russian Wikipedia dumps for training, we modify the wikifil.pl script such that it captures relevant characters and replaces all digits with relevant words in each language. For Genman, we add äöijß to the set of Latin characters. For Russian, we extract Cyrillic characters. Then, we manually truncate the parsed texts billion characters, and further truncate to the last complete word in the resulting text. For example, if the German enumeration was cut at the billion boundary, such that
12

---

Under review as a conference paper at ICLR 2021
Table 3: Detailed results of MUSE tests for fastText when training on English Wikipedia corpus. Higher is better. The results are obtained from different training run, hence slight difference from the main results.
algorithmic variant
sccuracy
YP
SIMLEX
MTark
RG
VERB
SEME VAL17
MTukk -287
MC
RW-STA
WS-353
TR.
WS-353
WS-353
-130
-999
-771
-143
-30
NFORD
-SIM
-3k
-ALL
-REL
SG.original
0.532
0.758
0.470
0.765
0.449
0.725
0.665
0.675
0,718
0.666
0.762
0.760
0,357
Our
SG_code_op
0.520
0.785
0.480
0.819
0.443
0.716
0.673
0.691
0.737
0.665
0.776
0.764
0.366
SG_no_subword
0.813
0.842
0.416
0.730
0.672
0.687
0.796
0.752
SG.batch
0.463
0.772
0,462
0.768
0.424
0.717
0.678
0.685
0,728
0.656
0.735
0.767
0.364
SG NS CT
0.450
0.739
0.448
0.660
0.481
0.685
0.662
0.636
0.680
0.622
0.689
0.735
0.338
SG_BPE_20K
0.468
0.793
0,409
0.842
0.431
0.711
0.658
0.692
0,749
0.671
0.795
0,750
0.376
SG_BPE_40K
0.547
0,814
0.404
0.810
0.411
0.721
0.658
0.706
0,760
0.675
0.754
0.759
0.365
200K
0.788
0.429
0.712
0.672
0.686
0.783
0.756
CBOW original
0.364
0.610
0.403
0.738
0,374
0.662
0.570
0.494
0.543
0.563
0.711
0.706
0.367
Our work:
CBOW pop
0.350
0.604
0.404
0.701
0.377
0.663
0.585
0.508
0.546
0.568
0.698
0.706
0.365
CBOW su_subd
0.435
0.706
0.663
0.749
CBOW NS II
0.368
0.607
0.391
0.682
0.379
0.669
0.594
0.513
0.547
0.572
0.681
0.706
0.353
CBOW 80
0.378
0.657
0,412
0.700
0.384
0.643
0.652
0,611
0.667
0.694
CBOW I60
0,369
0,683
0,408
0.632
0,386
0.641
0.656
0,628
0.645
0.598
0,590
0.667
0,293
CBOW DH
0.426
0.677
0.416
0.731
0,383
0.691
0.653
0.565
0.617
0.616
0.715
0.731
0.358
CBOW DIINSS _S_
0.425
0.640
0.395
0.742
0,364
0.668
0.640
0.539
0,583
0.603
0.699
0.718
0,340
CROW_DR DH_N__HNN 80
0.344
0.703
0,418
0.709
0.364
0.648
0.678
0.658
0.667
0.632
0.627
0.710
0.319
CBOW DIL.NS.166
0.284
CROW DHF
0.466
0.751
0.514
0.820
0.449
0.691
0.624
0.653
0,710
0.676
0.792
0.737
0.386
CBOW DHF NS. II
0.452
0.753
0,481
0.791
0,345
0.666
0.653
0.704
0.726
0.647 0.740
0.731
CBOW DDH, .SS800
0.366
0.736
0,418
0.743
0.346
0.636
0.637
0.695
0,701
0.627 0.636
0.705
0,320
CHOW DHI I68
0.276
0.669
0.398
0.704
0,350
0.567
0.601
0.673
0.670
0.579 0.599
0.664
0.283
CBOW BPE_20K
0.35
0.767
0,408
0.808
0.432
0.689
0.668
0.650
0,714
0.648
0.734
0.712
0.367
CBOW_RP _PE_
0.354
0.787
0.373
0.818
0.439
0.702
0.661
0.664
0.727
0.649 0.711
0.712
0.372
CBOW BO0K
0.385
0.758
0.738
0.414
0.716
0.672
0.724
eins zwei drei vier
becomes
eins zwei dr truncate it to
eins zwei Finally, we use iconv ensure UTF-8 format.
For example, for the Russian Wikipedia dump, the order of actions is:
wget <wiki dump address>/<ru.dd dump>
perl wikifil-ru. <ru. dumpp > ruwiki head -a 1000000000 ruwiki ruwiki9 manually truncate the text to the last complete word iconv -t utf-8 ruwiki9 -o ruwikig-ure
We train the embeddings on the file ruwiki9-utÉ.
D COMPATIBILITY WITH HUGGING FACE TOKENIZERS
We implement tokenization in way that it is compatible with the files produced by BypeelevelBPETToeenizerr in the Hugging Face Tokenizers library We apply the same char-
13

---

Under review as a conference paper at ICLR 2021
Table 4: Detailed results of MUSE tests for fastText when training on German Wikipedia corpus. Higher is better. The results are obtained from different training run, hence slight difference from the main results.
algorithmic
accuracy
variant
ZG222
GUR65
SIMLEX-999
GUR350
SEMEVALI7
WSS-33
SG oriigiaa
0.339
0.588
0.372
0.726
0.659
0.730
Our work:
SG code opt
0.373
0.617
0.371
0.724
0.699
0.716
SG .o._wwubwordd
0.395
0.363
0.718
0.677
0.796
SG batch
0.372
0.627
0.418
0.733
0.700
0.747
SG SG_NS.CK WFS.CT
0,332
0,600
0,388
0.702
0.688
0.700
SG_BPE_20K
0.387
0,632
0.358
0.720
0.692
0.703
SG.BPE_40K
0,395
0,644
0,385
0.722
0.707
0.749
SG_BPE_20OK
0.385
0.638
0.327
0.714
0.691
0.772
CBOW _original
0,284
0,517
0.308
0.695
0.570
0.701
Our work:
CROW _code_opt
0.284
0.517
0.319
0.693
0.579
0.695
CBOW subword
0.357
0.627
0.363
0.709
0.658
0.672
CBOW NS 11
0.310
0.525
0.302
0.690
0.596
0.690
CBOW NS
0.320
0.534
0.372
0.633
0.659
0.600
CBOW 160_
0.307
0.480
0.356
0.608
0.610
0,511
CBOW DH
0.313
0,543
0,344
0.709
0,618
0.672
CBOW DH_NS_I
0.324
0,541
0.322
0.690
0.624
0.687
CBOW DHNNS 80
0.332
0.546
0.388
0.631
0.678
0.561
CBOW DH_DDHSNSIEE 160
0.282
0.469
0,291
0.601
0.590
0,510
CBOW DHF
0.387
0.642
0.442
0.695
0.710
0.675
CBOW DHF NS
0.376
0.647
0.417
0.664
0.724
0.699
CBOW DHF NS 80
0.339
0.540
0.406
0.591
0.619
0.587
CBOW DHF 160
0.267
0.441
0,276
0.572
0.524
0.496
CBOW BPE
0.319
0.620
0.361
0.689
0.660
0.622
CBOW _BPE_4KK
0.338
0.628
0.388
0.709
0.650
0,607
CBOW 2POK
0.343
0.649
0.332
0.713
0.660
0.684
acter mapping for UTF-8 characters that use more than one byte, and preprocess the words from the vocabulary such that each begins with a special delimiter character G.
Note that the number of tokens h must be selected during tokenization with the Tokenizers library.
bpe ByceleveeLBPTToeeiieerr bpe. teeettrii file>],
vocab_size=<hh>
bpe. save (<path>, <filename>)
(Note the []
brackets.)
E
How TO RUN EXPERIMENTS
We provide parameterized code to replicate our experiments. It is modification of the original fastTexx library Note that we disabled the production of bin file to reduce saving time and save storage space. Our experiments apply to unsupervised training with skip-gram and CBOW.
In order to compile, CMake, Intel ICPC compiler and CPU with AVX-512 support are required. Please compile with:
cmake make
"https//gituu..co//TSStatb
14

---

Under review as a conference paper at ICLR 2021
To run, please the command:
fasttext (cbow, skipgram) -input <corpus file> -output <embeddings file>
<arguments>
Selecting code optimizations. To run a particular algorithm mode, use the argument -mode <mode>. Table 5 explains all available modes.
Table An overview -mode arguments and their connection to our experiments. Parameter s is set with -saared 
.
argument
mode
-mode normal
_code_opt (default)
-mode batched
SG batch iskip-gram only)
SGLNS_CT (skip-gram) CBOW_NS, __Ww_NS_s (CBOW)
-mode
-mode
CBOW DH (CBOW only)
-mode dhf
CBOW DHF (CBOW only)
-mode dh-ns CBOW BROW_DHLNS.s (CBOW only)
-ooe dhf-ns SHOW_DHFARD (CBOW only)
By default, s (the number of words sharing negative samples) set to 2C 1, where is the maximum context window size (set in the original Test with the argument -ws <c>). To set different 8, use -shared 
, Setting this argument to zero will result the default setting.
Selecting BPE tokens no_subword. To use BPE tokens instead of subwords, provide paths to the merge and vocab files produced by the Tokenizers library.
-token-merges <path/to/E.txt -token-vocal <path/to/E.json
Both these arguments must be set to enable the BPE run. The number of tokens h is obtained from the merge and vocab files. For our experiments, we produce these files using our training corpora. Note that these two arguments incompatible with -no-subwords.
Finally, to run the subword (word2vee) version of our code, use the argument -no-subwords. Note that it is incompatible with -tth-ttoo--eeeee and -token-vocab.
We run both the BPE and no subword experiments using -mode normal (default). The remaining arguments identical to those used by the original fastText code.
To obtain the results for SG-original and CBOW_origing please run the original library.
F COMPLETE RESULTS: ENGLISH, RUSSIAN AND GERMAN
We provide complete results for the embeddings trained on English (Tab. 6. German (Tab. 7 and Russian (Tab. [8 corpora. In negative sharing (NS), we use = 11, 80, 160.
G VARIANCE OF RESULTS
Due to low stability of word embeddings across different trainings (e.g.. (Antoniak & Mimno, 2018), we perform multiple experiments on our variants to characterize standard deviations ob- tained for the scores of word analogy task the questions-words test set (QW). We measure across
15

---

Under review as a conference paper at ICLR 2021
Table 6: Full results: accuracy, runtime and speedup achieved with our library over fastText when training on English Wikipedia corpus. Blue: best accuracy in category, bold: Pareto-optimal accu- racy, speedup is over the original fastText run with the same number of threads, and scaling the speedup of 20 threads vs. thread for our code. For accuracy, speedup and scaling, higher is better. For time, lower is better.
algurithmic
accurccy
time (8)
speeduu (times)
scaling
variant
QW
BATS
Battig
20
20
thread
threads
thread
thrends
SGooigginal
41.54
Our work:
SG.codc.opt
26.37
63.75
9.29
41.61
0.647
40.85
9088
810
3.7
11.2
SG subword
47.42
48.07
12.97
27.81
0.635
41.27
3309
253
10,2
13.1
SG batch
24.28
62.47
8.92
41.84
0.630
40.76
7631
625
4,4
3.6
12.2
SG.NS.CT
60.52
6.74
0.611
257
SG (GGBPE,20R
40.78
53.85
12.86
34,50
0.642
40.43
6843
527
4.9
4.2
13.0
SG_BPE
SG_BPE_200K
54.98
44.79
12.65
27.73
0.634
41.46
7041
526
4.8
4.2
13.4
CBOW CBOW_original
9.27
67.74
5.53
63.42
0.546
32.98
19614
1484
1.0
13.2
Our work:
CBOW code. upp.
CROW no suhword
$1.00
54.49
14.84
32.29
0.614
40.68
951
73
20.6
20.4
13.1
CBOW NSS.1
7.18
63.30
4.96
60.27
0.545
32.63
3623
5,4
CBOWLNSS
6.11
63.41
4.70
60.27
0.570
33.74
3604
604
5.4
2.5
CBOW NS_160
5.10
62.50
4.41
60.85
0.555
30.74
3584
602
5.5
2.5
CBOW_DR
11.38
68.73
6.51
63,58
0.583
36.30
4079
555
4.8
7.4
CBOW DS_NS_
9.19
64.19
6.03
58.77
0.564
34.66
3422
480
5.7
3.1
7.1
5.9
CBOW DH NS_60
3.90
54.84
3.77
49.
0.536
29.44
3371
454
5.8
3.3
7.4
CBOW DHF
5.29
57.44
3.94
51.71
0.633
31,37
4100
446
4,8
CBOW DHIINS.I1
2.71
44.51
2.75
43.38
0.608
3214
398
6.1
CBOW DUE NS. 80
2.84
47.19
3.10
43.26
0.582
29.65
3136
392
6.3
38
CROW DWF_NH__IRR _S0
2.00
52.77
2.63
53.79
0.539
24.78
3132
392
6.3
CBOW BPE 20K
22.65
46.01
10.17
36.98
0.607
35.56
1717
135
11.4
11.0
12.8
40.47
CBOW CBOW_BPE_200 2O0K
43.33
26.40
8.12
24.70
0.606
40.39
1714
131
11.4
11.3
13.0
5 runs. We were unable perform more tests due to time constraints, but the results provide hints on stability. We present the results Table 9 We note that the standard deviation rarely increases above 1. Moreover, it is more likely to be high for no subword and BPE variants of the code.
16

---

Under review as a conference paper at ICLR 2021
Table 7: Full results: accuracy, runtime and speedup achieved with our library fastText when training on German Wikipedia corpus. Blue: best accuracy in category, bold: Pareto-optimal accu- racy, speedup is over the original fastText with the same number of threads, and scaling is the speedup of 20 threads vs. thread for our code. For accuracy, speedup and scaling, higher is better. For time, lower is better.
algorithmic
accuracy
time (s)
speedup (times)
scaling
variant
QW MUSE
20
20
sem.
syn.
thread
threads
thread
threads
SG.original
21.13
49,94
0.587
51747
6095
1.0
8.5
Our work:
SG_codc_op
19.41
49.12
0.575
14599
1233
3.5
4.9
11.8
SG no__no.ru
subword SG_no_subword
42.31
27.29
0.589
4511
325
11.5
18.8
13.9
SG. batch
17.43
49.13
0.573
11249
870
7.0
12.9
SG SG.NS_C CT
7.08
47.41
0.547
4506
383
11.5
15.9
11.8
SG (G.BPE.20K
29.91
36.83
0.589
9440
682
5.
8.94
13.9
SG [G_BPE.40K
34.55
29.96
0,593
9908
678
5.2
9.0
14.6
SG BPP..2000
45.90
23.39
0.593
9830
675
5.3
9.0
14.6
CBOW original
4.71
58.94
0.507
31674
3837
1.0
1.0
8.3
Our work:
CBOW code opt
4.30
59.09
0,510
7452
CBOW _n__subwrrd
37.75
27.66
0,559
1315
92
24.1
41.8
14.3
CBOW _NS11
3.17
55.67
0.520
5754
1050
5.5
3.7
5.5
CBOW NS_80
3.04
52.50
0.515
5664
1068
5.6
3.6
5.3
CBOW NNS. 160
1.58
46.71
0.472
5615
1045
5.6
5.4
CBOW DH
5.96
59.81
0.527
6406
911
4.2
7.0
CBOW DS 11
4.57
55,56
0.530
S532
841
5.7
4.6
6.6
CBOW DH .8
2.30
50.95
0.522
5812
831
5.5
4.6
7.0
CBOW DH_NS_160
1.51
45.25
0,450
6097
829
S.2
4.6
7.4
CBOW DHF
1.86
54.17
0.591
6117
845
7.2
CBOW DHF NS_11
0.82
46.36
0,562
5049
751
6.3
5.1
6.7
CBOW DHF NS_80
1.03
46.97
0.498
5028
737
6.3
5.2
6.8
CBOW WWF_DHF_MT WWW_DHF_NS_OG 160
1.08
41.07
0.420
5059
764
6.3
5.0
6.6
CBOW _BF_20K
11.19
31.31
0.542
2413
182
13.1
21.1
13.3
CROW 40K
18.67
23.25
0.557
2389
174
13.3
22.0
13.7
CBOW BPE 200K
27.81
16.14
0.560
2461
170
12.9
22.6
14.5
17

---

Under review as a conference paper at ICLR 2021
Table 8: Full results: accuracy, runtime and speedup achieved with our library fastText when training on Russian Wikipedia corpus. Blue: best accuracy in category, bold: Pareto-optimal accu- racy, speedup is over the original fastText with the same number of threads, and scaling is the speedup of 20 threads vs. thread for our code. For accuracy, speedup and scaling, higher better. For time, lower is better.
algorithmic
accuracy
time (8)
speedup (times)
scaling
variant
QW
20
20
MUSE
scm.
syn.
thread
threads
thread
threads
SG.original
12.29
77.61
0.633
30219
2959
1.0
10.2
Our work:
SG_codc_op
12,86
77.81
0.622
7409
710
4.1
4.2
10.4
SG_no_ SG.no.subword nuuuuoo
23.82
42.89
0.588
2501
179
12.1
16.5
14.0
SG. batch
10.58
76.32
0.629
5914
565
5.1
5.2
10.5
SG
5.58
77.81
0.602
2289
254
13.2
11.7
9.0
SG (G.BPE_20K
14.22
51.86
0.621
5179
376
5.8
7.9
13.8
SG 25_BPE.40K
16.75
46.81
0.608
$212
374
5.8
7.9
13,9
SG (G.BPE.20 200K
22.90
45.91
0.600
5230
370
5.8
8.0
14.1
CBOW original
8.88
80.78
0.495
20510
2599
1.0
1.0
7.9
Our work:
CBOW code opt
9.18
79.79
0.497
4541
569
8.0
CBOW no_swworrd
16.01
37.35
0.532
731
51
28.1
50.5
14.2
CBOW 1NS_11
7.92
77.32
0.493
4166
519
4.9
5.0
8.0
CBOW _NS_00
7,46
75.83
0.490
4105
518
5.0
5.0
7.9
CBOW .NS. 160
6.61
68.75
0.391
4311
518
4.8
5.0
8.3
CBOW DH
9.17
81.43
0.523
4790
455
5.7
10.5
CBOW DH 11
7.82
78.95
0.513
4179
403
4.9
6.5
10.4
CBOW DH.NS 80
7.41
75.63
0.449
3969
403
5.2
6.4
9.8
CBOW WWW.DH.NS.16.
6.23
66.47
0.368
3796
405
5.4
6.4
9.4
CBOW DHF
7.32
78.75
0.557
4451
418
6.2
10.6
CBOW DHF_NS_1L
4.79
73.60
0,548
3400
366
6.0
7.1
9.3
CBOW DHF NS 80
5,50
66.22
0.458
2866
368
7.2
7.1
7.8
CBOW DHF_NS_16
5.68
60.92
0.384
3024
366
6.8
7.1
8.3
CBOW WOO_LPF_220 BPE_20K
8.28
34.82
0.513
1371
103
15.0
25.2
13.3
CBOW BPE 40K
8.96
40.66
0.493
1349
100
15.2
26.1
13.6
CBOW BPE 200K
11.13
35.96
0.537
1343
96
15.3
27.0
14.0
18

---

Under review as a conference paper at ICLR 2021
Table 9: Standard deviation of the score obtained with word analogy task on QW. Lower is better.
English
German
Russian
algorithmic
variant
syn.
syn.
syn.
SG SGoiiginall
0.28
0.28
0.43
0.36
0.33
0.83
Our work:
SG_code_ops
1,28
0.76
0.39
1.24
SG_no.subword
1.10
0.76
0,96
0.66
0.55
1.82
SG_batch
0.51
0.33
1.54
0.52
0.40
0.98
SG NS CT
0.39
0.51
0.69
0.88
0.25
0.95
SG BPE 20K
1.51
0.28
0.76
0.38
0.14
0.23
BPE 40K
1.04
0.50
0.33
0.48
SG_BPE_2006
1.93
0.54
0.84
0.64
0.36
2.46
CBOW Loriginil
0.13
0.23
0.10
0.21
0.08
0.63
Our work:
CBOW code.opt
0.07
0.19
0.11
0.35
0.11
0.47
CBOW subword
1.04
0.37
0.38
0.29
0.32
0.74
CBOW NS 11
0.18
0.48
0.08
0.57
CBOW _S_
0,30
0.41
0.09
0.29
0.11
0.72
CBOW _S__16
0.13
0.29
0.11
0.37
0.10
0.47
CBOW HDH
0.34
0.33
0.11
0.40
0.12
0.44
CBOW DH NS
0.10
0.40
0.17
0.39
0.21
1.25
CBOW DH NS
0.30
0.38
0.20
0.42
0.24
0.53
CBOW DH NS. 160
0,26
0.57
0.06
0.25
0.17
0.91
CBOW DHF
0.31
0.24
0.05
0.38
0.14
0.81
CBOW WWW.DHFF.LD 1HH.NN.11
0.13
0.75
0.13
0.63
0.17
1.35
CBOW DHF WIFNSS.RR NS_80
0.19
0.76
0.06
0.52
0.14
0.60
CBOW DHF NN_.NS166 160
0.12
0.55
0.07
0.23
0.14
1.06
CBOW BPE_20K
0.27
0.25
0.24
0.38
0.23
0.69
CBOW 4PE_40K
0.45
0.37
0.48
0.31
0.41
0.46
CBOW BPE 200K
0.24
0.28
0.46
0.14
0.10
1.28
19


Reviews:
[{"review_id": "nuYD5J0JDmX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a few implementation tricks to improve the performance of word2vec and fastText algorithms on CPUs. The authors offer an implementation that is optimized with intrinsics. In addition to that, the authors discuss a few different implementational improvements that can be made at the algorithmic level to improve the training speed of fastText.\n\nPros:\n- The paper is easy to read and follow\n- The authors have tried a variety of implementation techniques aimed at improving training performance\n- The experimental results shows large speedups on CPUs in a few cases without trading off accuracy.\n\nCons:\nThe novelty as well as the importance of this work is not clear. The code optimizations mentioned in the paper using AVX-512 intrinsics targeted at Intel CPUs, blocking and merging, or load/store optimizations of hidden layer and gradients aren't particularly new and neither are they specific to fastText. The algorithmic implementation variants tried on the other hand such as ignoring subwords or batching for skip-grams are specific to fastText, but are fairly straight-forward. The only novel contributions to the algorithm that result in improvements appear to be dynamic hidden-layer update and byte-pair encoding. Additionally, the paper fails to motivate why this line of optimization is important while there appear to be other straight-forward ways to make the training significantly faster (distributed training, clean GPU implementation that is open-source, etc).", "extra_scores": {}, "extra_text": {}}, {"review_id": "cqcTcEltAut", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the approaches that reduce the training times of word2vec and fastText. To improve the efficiency, it uses several techniques such as negative sample sharing, batched updates, and a byte-pair encoding-based alternative for subword units. By using English, German, and Russian languages data, it shows that the proposed approach is much faster than the existing approaches. \n\nBasically, I like the motivation and results of the paper; it is quite an important research problem to reduce training times of popular word embedding approaches. Besices, this paper is well structured and easy to follow; contributions are clear and proposed approaches are well described. However, the approaches proposed in the paper are somewhat technically shallow. I think more technical depth is needed for ICLR papers. This paper seems to be an engineering or industrial paper rather than a research paper; ICLR is not the best venue for the paper. To improve the paper, it is good to clearly describe the highlight of the proposed approach and explicitly show its theoretical property. While several approaches are described in Section 3, I think one or two approaches with detailed technical analyses are enough.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZXEJ1req-_N", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies several algorithmic and code optimization techniques to reduce the training time for word2vec and fastText. The final implementation runs 3-20x times faster than the baselines on manycore CPUs, with almost no quality loss. The improved implementation is a result of the combination of many techniques, including namely no_subword, Minibatching, Negative sharing etc. \n\n\n*Strong points*\n- The 3-20x speedup is impressive.\n- Given the popularity of word2vec and fastText, the open-sourced version of the proposed method might benefit many applications.\nThe experiment is solid.\n\n*Weak points*\n- This paper is more like a technical report, describing the details of an improved implementation. Many of the optimization techniques being used can also be found in previous works. Therefore the novelty of this paper might be slightly below the standard of ICLR.\n- This paper is hard to follow, partly because of the organization of this paper. It would be beneficial to address the following potential issues,\n     +  Two many engineering details are included. Consider moving some of those details into appendix, and include more high level intuition why those optimization should be included.\n     + Better to include an overview of techniques before explaining the detailed implementation. This helps the audience quickly understand the major contributions/novelties compared with existing works.\n     + The main text frequently refers to the line# of the source code, which hurts the readability of this paper. \nIdeally, provide a table to summarize the datasets being used in this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PjF6uZQZW3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The current work introduces a new approach that speeds up the training process of word embedding models such as word2vec and fasttext on specific hardware.\nThe improvements are enumerated as follows: Code performance optimizations, using AVX-512 extensions, suppression of subword information (only for fasttext), modifying batch over training, Negative sampling shared for consecutive words, Dynamically updating the hidden layer and the use of Byte-Pair vocabulary.\nTests are conducted on a set of word based benchmarks in many languages, where improvements in terms of both performance and training time is shown, being CBOW_no_subword the best configuration in terms of speed up across languages.\n\nBy and large the current work is ok. The authors presented in a clear way what is the current state, the models they build on top of, the techniques used and present experimentation that show the effectiveness of such improvements.\nMy main concern is, on the experimentation, where I would have chosen more standard datasets. I guess WordSim 353 is a must for a semantic similarity evaluation and its multilingual extension. From what I saw in the appendix, MUSE is an average of many datasets, which includes WS353.\nIn addition, a discussion or application of the author's improvements on GloVe seems also reasonable. \nDespite that, I think the submission is strong.\nSome minor recommendations below.\n\nEvaluation: Averaging on several monolingual datasets is ok, if you present the individual scores as well. In fact, columns showing almost no difference between models. I would recommend to include some individual starts, for instance WS353, SimLex999, which have also multilingual counterparts from Leviant & Reichart (2015)\n\nTable 1/2/3: Include a comment (or an arrow) to clarify if the scores are higher (or lower) the better.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work describes a series of strategies for optimizing the training speed of
word embeddings (as in word2vec and fasttext).

All reviewers appreciate the convincing empirical results, which are without a
doubt impressive.  Reviewers also mostly agree that speeding up embedding
training is important, and there is no doubt that this type of paper is
appropriate for ICLR (as clearly highlighted in the CfP.)

However, the specific optimization strategies deployed and described here
are deemed not to bring novel insight, useful in itself to the community, beyond the
software contribution described.
The paper seems to mostly serve as documentation of the
implementation, limiting its value and impact to further research.
The pedagogic value is also limited, as the paper tackles multiple different,
eclectic optimizations, a narrative strategy that does not leave room to describe a single one more
generally, helping the community find other places to apply it.
All in all this leads to a borderline negative assessment, and I cannot
recommend acceptance.
Decision: Reject