Title: Riemannian Optimization for Skip-Gram Negative Sampling
Authors: Alexander Fonarev
Alexey Grinchuk
Gleb Gusev
Pavel Serdyukov
Ivan Oseledets
Venue: ICLR 2017
Keywords: Natural language processing
Unsupervised Learning

Abstract: Skip-Gram Negative Sampling (SGNS) word embedding model, well known by its implementation in "word2vec" software, is usually optimized by stochastic gradient descent. It can be shown that optimizing for SGNS objective can be viewed as an optimization problem of searching for a good matrix with the low-rank constraint. The most standard way to solve this type of problems is to apply Riemannian optimization framework to optimize the SGNS objective over the manifold of required low-rank matrices. In this paper, we propose an algorithm that optimizes SGNS objective using Riemannian optimization and demonstrates its superiority over popular competitors, such as the original method to train SGNS and SVD over SPPMI matrix.

TL;DR: We train word embeddings optimizing Skip-Gram Negative Sampling objective (known by word2vec) via Riemannian low-rank optimization framework

Review discussion:
Does using Riemannian optimization allow the model to converge faster than the alternatives?
I'm asking this because the evaluation doesn't show a dramatic advantage to RO-SGNS; the 1% difference on the word similarity benchmarks is within the range of hyperparameter effects (see "Improving Distributional Similarity with Lessons Learned from Word Embeddings", (Levy et al., 2015)). The proposed method is elegant from a theoretical perspective, but I was wondering if there are also some tangible benefits to this approach.
Thank you for your question!
The goal of any SGNS optimization method (Step 1 from the introduction section) is to find a good solution in terms of SNGS objective. From that point of view, our approach achieves the significant improvement (see Table 1).
Moreover, the improvement on Step 1 entails the improvement on Step 2 (see Table 2).
So, all experimental results together show that our approach makes sense, what, most importantly, opens the way to applying even more advanced approaches based on the proposed two-step decomposition of the problem (e.g., more advanced Riemannian optimization techniques for Step 1 or a more sophisticated treatment of Step 2).
I still need to read the paper more carefully. My main concern at this moment is what is the main contribution of this paper.
1. Does the current paper have significant novelty in terms of optimization, or is it applying relatively mature techniques to the low-rank optimization problem of SGNS?
2. Is the main contribution about properly optimization the SGNS objective, which was previously done only approximately by SVD? In that case, it is easy to believe that the paper can obtain improvement over SVD factorization.
3. Have you compared with other popular optimization algorithms in machine learning for the SGNS objective, such as gradient descent/block coordinate descent or CG/L-BFGS? I am not asking for all comparison; and it appears the authors compared with SGD in the experiments.
But if the contribution is mainly proper optimization, it makes sense to compare different algorithm in terms of efficiency (e.g., it would be good to show some learning curves). A comparison with gradient descent would be informative as we will be able to see the effect of Euclidean geometry vs Riemannian manifold geometry.
Thank you for the questions!
1. The paper's novelty is in the demonstration that the word embeddings learning consists of two separate steps (see Section 1). Step 1: searching for a matrix with a low-rank constraint that is good in terms of SGNS objective, and Step 2: searching for a good factorization of that matrix in terms of the target linguistic quality metric of word embeddings. Unfortunately, most previous approaches mixed these two steps into a single one, what entails a not completely correct formulation of the optimization problem (see Sections 1 and 2.2.2). In our paper, we demonstrated that the use of this two-step reformulation may increase the quality of embeddings.
As an example, we showed that the proper optimization over low-rank matrix X=WC instead of optimization over factors W and C independently increases the quality of Step 1 (see Table 1).
Of course, Step 2 may be improved as well, but we regard this as a direction of future work. Hopefully, this two-step perspective will be helpful for future research on the topic of word embeddings learning. We will highlight the ideas described above in the next revision of the paper in order to make our contribution more clear.
2. Yes, our Riemannian optimization based method outperforms the method based on SVD according to our experiments.
That is what we expected to gain from the reformulation of the embeddings learning problem into two steps with clear objectives.
3. We have also experimented with the blockwise alternating optimization over factors W and C. The alternating minimization led to almost the same results as SGD, and both of these techniques appeared to be worse than the Riemannian optimization based method. We decided not to include the alternating minimization results into the paper because the goal of the paper is to demonstrate the superiority of two-step problem reformulation, and not necessarily to compare all most advanced optimization methods in terms of SGNS minimization (nevertheless, more advanced Riemannian optimization techniques significantly outperform advanced component-wise methods in similar tasks, e.g. see [1]).
That is why we compared the proposed method only to the most popular one based on SGD. Anyway, we implemented the alternating minimization and we will highlight that the alternating minimization approach is outperformed, as well as SGD and SVD, in the next revision of the paper.
[1] Tan, Mingkui, et al. "Riemannian Pursuit for Big Matrix Recovery." ICML 2014.
This paper presents a principled optimization method for SGNS (word2vec).
While the proposed method is elegant from a theoretical perspective, I am not sure what the tangible benefits of this approach are.
For example, does using Riemannian optimization allow the model to converge faster than the alternatives? The evaluation doesn't show a dramatic advantage to RO-SGNS; the 1% difference on the word similarity benchmarks is within the range of hyperparameter effects (see "Improving Distributional Similarity with Lessons Learned from Word Embeddings", (Levy et al., 2015)). The theoretical connection to Riemannian optimization is nice though, and it might be useful for understanding related methods in the future.
Thank you for your reply! I have commented it in the answer to your other message.
Dear authors,
The authors' response clarified some of my confusion.
But I still have the following question:
-- The response said a first contribution is a different formulation: you divide the word embedding learning into two steps, step 1 looks for a low-rank X (by Riemannian optimization), step 2 factorizes X into two matrices (W, C). You are claiming that your model outperforms previous approaches that directly optimizes over (W, C).
But since the end result (the factors) is the same, can the authors provide some intuition and justification why the proposed method works better?
As far as I can see, though parameterized differently, the first step of your method and previous methods (SGD) are both optimizing over low-rank matrices.
Admittedly, Riemannian optimization avoids the rotational degree of freedom (the invertible matrix S you are mentioning in sec 2.3), but I am not 100% certain at this point this is the source of your gain; learning curves of objectives would help to see if Riemannian optimization is indeed more effective.
-- Another detail I could not easily find is the following. You said a disadvantage of other approaches is that their factors W and C do not directly reflect similarity. Did you try to multiply the factors W and C from other optimizers and then factorize the product using the method in section 2.3, and use the new W for your downstream tasks? I am not sure if this would cause much difference in the performance.
Overall, I think it is always interesting to apply advanced optimization techniques to machine learning problems. The current paper would be stronger from the machine learning perspective, if more thorough comparison and discussion (as mentioned above) are provided.
On the other hand, my expertise is not in NLP and I leave it to other reviewers to decide the significance in experimental results.
The paper considers Grassmannian SGD to optimize the skip gram negative sampling (SGNS) objective for learning better word embeddings. It is not clear why the proposed optimization approach has any advantage over the existing vanilla SGD-based approach - neither approach comes with theoretical guarantees - the empirical comparisons show marginal improvements.
Furthermore, the key idea here - that of projector splitting algorithm - has been applied on numerous occasions to machine learning problems - see references by Vandereycken on matrix completion and by Sepulchre on matrix factorization.
The computational cost of the two approaches is not carefully discussed.
For instance, how expensive is the SVD in (7)? One can always perform an efficient low-rank update to the SVD - therefore, a rank one update requires O(nd) operations. What is the computational cost of each iteration of the proposed approach?
Thank you for your reply!
We think that the contribution of our paper is not only in the new application of the existing Riemannian optimization technique but also in the clear reformulation of the word embedding learning problem (see previous comments for details).
Speaking of the computational complexity, you are right — the complexity of our algorithm is not perfect and the algorithm can not be applied to large-scale datasets with millions of unique tokens in the training corpus. We are going to improve the efficiency of the algorithm in our future work.

Meta-review:
role: PC
data: comment: The paper is mostly clearly written. The observation made in the paper that word-embedding models based on optimizing skip-gram negative sampling objective function can be formulated as a low-rank matrix estimation problem, and solved using manifold optimization techniques, is sound. However, this observation by itself is not new and has come up in various other contexts such as matrix completion. As such the reviewers do not see sufficient novelty in the algorithmic aspects of the paper, and empirical evaluation on the specific problem of learning word embeddings does not show striking enough gains relative to standard SGD methods. The authors are encouraged to explore complimentary algorithmic angles and benefits that their approach provides for this specific class of applications.

Decision: Reject