Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The current work introduces a new approach that speeds up the training process of word embedding models such as word2vec and fasttext on specific hardware.
The improvements are enumerated as follows: Code performance optimizations, using AVX-512 extensions, suppression of subword information (only for fasttext), modifying batch over training, Negative sampling shared for consecutive words, Dynamically updating the hidden layer and the use of Byte-Pair vocabulary.
Tests are conducted on a set of word based benchmarks in many languages, where improvements in terms of both performance and training time is shown, being CBOW_no_subword the best configuration in terms of speed up across languages.

By and large the current work is ok. The authors presented in a clear way what is the current state, the models they build on top of, the techniques used and present experimentation that show the effectiveness of such improvements.
My main concern is, on the experimentation, where I would have chosen more standard datasets. I guess WordSim 353 is a must for a semantic similarity evaluation and its multilingual extension. From what I saw in the appendix, MUSE is an average of many datasets, which includes WS353.
In addition, a discussion or application of the author's improvements on GloVe seems also reasonable. 
Despite that, I think the submission is strong.
Some minor recommendations below.

Evaluation: Averaging on several monolingual datasets is ok, if you present the individual scores as well. In fact, columns showing almost no difference between models. I would recommend to include some individual starts, for instance WS353, SimLex999, which have also multilingual counterparts from Leviant & Reichart (2015)

Table 1/2/3: Include a comment (or an arrow) to clarify if the scores are higher (or lower) the better.


review_title: Official Blind Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper describes a few implementation tricks to improve the performance of word2vec and fastText algorithms on CPUs. The authors offer an implementation that is optimized with intrinsics. In addition to that, the authors discuss a few different implementational improvements that can be made at the algorithmic level to improve the training speed of fastText.

Pros:
- The paper is easy to read and follow
- The authors have tried a variety of implementation techniques aimed at improving training performance
- The experimental results shows large speedups on CPUs in a few cases without trading off accuracy.

Cons:
The novelty as well as the importance of this work is not clear. The code optimizations mentioned in the paper using AVX-512 intrinsics targeted at Intel CPUs, blocking and merging, or load/store optimizations of hidden layer and gradients aren't particularly new and neither are they specific to fastText. The algorithmic implementation variants tried on the other hand such as ignoring subwords or batching for skip-grams are specific to fastText, but are fairly straight-forward. The only novel contributions to the algorithm that result in improvements appear to be dynamic hidden-layer update and byte-pair encoding. Additionally, the paper fails to motivate why this line of optimization is important while there appear to be other straight-forward ways to make the training significantly faster (distributed training, clean GPU implementation that is open-source, etc).
review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper applies several algorithmic and code optimization techniques to reduce the training time for word2vec and fastText. The final implementation runs 3-20x times faster than the baselines on manycore CPUs, with almost no quality loss. The improved implementation is a result of the combination of many techniques, including namely no_subword, Minibatching, Negative sharing etc. 


*Strong points*
- The 3-20x speedup is impressive.
- Given the popularity of word2vec and fastText, the open-sourced version of the proposed method might benefit many applications.
The experiment is solid.

*Weak points*
- This paper is more like a technical report, describing the details of an improved implementation. Many of the optimization techniques being used can also be found in previous works. Therefore the novelty of this paper might be slightly below the standard of ICLR.
- This paper is hard to follow, partly because of the organization of this paper. It would be beneficial to address the following potential issues,
     +  Two many engineering details are included. Consider moving some of those details into appendix, and include more high level intuition why those optimization should be included.
     + Better to include an overview of techniques before explaining the detailed implementation. This helps the audience quickly understand the major contributions/novelties compared with existing works.
     + The main text frequently refers to the line# of the source code, which hurts the readability of this paper. 
Ideally, provide a table to summarize the datasets being used in this paper.

review_title: impressive speedup but not novel enough

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes the approaches that reduce the training times of word2vec and fastText. To improve the efficiency, it uses several techniques such as negative sample sharing, batched updates, and a byte-pair encoding-based alternative for subword units. By using English, German, and Russian languages data, it shows that the proposed approach is much faster than the existing approaches. 

Basically, I like the motivation and results of the paper; it is quite an important research problem to reduce training times of popular word embedding approaches. Besices, this paper is well structured and easy to follow; contributions are clear and proposed approaches are well described. However, the approaches proposed in the paper are somewhat technically shallow. I think more technical depth is needed for ICLR papers. This paper seems to be an engineering or industrial paper rather than a research paper; ICLR is not the best venue for the paper. To improve the paper, it is good to clearly describe the highlight of the proposed approach and explicitly show its theoretical property. While several approaches are described in Section 3, I think one or two approaches with detailed technical analyses are enough.

review_title: Need technical depth

Meta-review: This work describes a series of strategies for optimizing the training speed of
word embeddings (as in word2vec and fasttext).

All reviewers appreciate the convincing empirical results, which are without a
doubt impressive.  Reviewers also mostly agree that speeding up embedding
training is important, and there is no doubt that this type of paper is
appropriate for ICLR (as clearly highlighted in the CfP.)

However, the specific optimization strategies deployed and described here
are deemed not to bring novel insight, useful in itself to the community, beyond the
software contribution described.
The paper seems to mostly serve as documentation of the
implementation, limiting its value and impact to further research.
The pedagogic value is also limited, as the paper tackles multiple different,
eclectic optimizations, a narrative strategy that does not leave room to describe a single one more
generally, helping the community find other places to apply it.
All in all this leads to a borderline negative assessment, and I cannot
recommend acceptance.
Decision: Reject