Title: Efficient Training on Very Large Corpora via Gramian Estimation
Authors: Walid Krichene
Nicolas Mayoraz
Steffen Rendle
Li Zhang
Xinyang Yi
Lichan Hong
Ed Chi
John Anderson
Venue: ICLR 2019
Keywords: similarity learning
pairwise learning
matrix factorization
Gramian estimation
variance reduction
neural embedding models
recommender systems

Abstract: We study the problem of learning similarity functions over very large corpora using neural network embedding models. These models are typically trained using SGD with random sampling of unobserved pairs, with a sample size that grows quadratically with the corpus size, making it expensive to scale.
We propose new efficient methods to train these models without having to sample unobserved pairs. Inspired by matrix factorization, our approach relies on adding a global quadratic penalty and expressing this term as the inner-product of two generalized Gramians. We show that the gradient of this term can be efficiently computed by maintaining estimates of the Gramians, and develop variance reduction schemes to improve the quality of the estimates. We conduct large-scale experiments that show a significant improvement both in training time and generalization performance compared to sampling methods.

TL;DR: We develop efficient methods to train neural embedding models with a dot-product structure, by reformulating the objective function in terms of generalized Gram matrices, and maintaining estimates of those matrices.

Review discussion:
This paper proposes a method for estimating non-linear similarities between items using Gramian estimation. This is achieved by having two separate neural networks defined for each item to be compared, which are then combined via a dot product. The proposed innovation in this paper is to use Gramian estimation for the penalty parameter of the optimization which allows for the non-linear case. Two algorithms are proposed which allow for estimation in the stochastic / online setting. Experiments are presented which appear to show good performance on some standard benchmark tasks.
Overall, I think this is an interesting set of ideas for an important problem. I have two reservations.
First, the organization of the paper needs to be addressed in order to aid user readability. The paper often jumps across sections without giving motivation or connecting language. This will limit the audience of the paper and the work.
Second (and more importantly), I found the experiments to be slightly underwhelming. The hyperparameters (batch size, learning rate) and architecture don’t have any rationale attached to them. It is also not entirely clear whether the chosen comparison methods fully constitute the current state of the art.
Nonetheless, I think this is an interesting idea and strong work with compelling results.
Editorial comments:
The organization of this paper leaves something to be desired. The introductions ends very abruptly, and then appears to begin again after the related work section. From what I can tell the first three sections all constitute the introduction and should be merged with appropriate edits to make the narrative clear.
“where x and y are nodes in a graph and the similarity is wheter an edge” → typo and sentence ends prematurely.
Thank you for your review and your helpful suggestions.
We updated the organization following the reviewer's suggestions, by reorganizing the introduction and improving the transitions between sections. We also added a comment about our choice of hyper-parameters: in the main experiments of Section 4, the hyper-parameters were cross-validated using the baseline. The effect of some of the hyper-parameters is further studied in the appendix: the effect of the batch size and learning rate is studied in Appendix D.2 (now Appendix E.4 in the revision), and the effect of the penalty coefficient λ is illustrated in Appendix C (now Appendix D in the revision). We did not include these results in the main body of the paper for space constraints, and to keep the message focused, but we added a note to Section 4 pointing to the appendix for further details on the effect of the various hyper-parameters.
This paper proposes an efficient algorithm to learn  neural embedding models with a dot-product structure over very large corpora. The main method is to reformulate the objective function in terms of generalized Gramiam matrices, and maintain estimates of those matrices in the training process. The algorithm uses less time and achieves significantly better quality than sampling based methods.
1.
About the experiments, it seems the sample size for sampling based experiments is not discussed. The number of noise samples have a large influence on the performance of the models. In figure 2, different sampling strategies are discussed. It would be cool if we can also see how the sampling size affects the estimation error.
2. If we just look at the sampling based methods, in figure 2a, uniform sampling’s Gramian estimates is the worst.
But the MAP of uniform sampling on validation set for all three datasets are not the worst. Do you have any comments?
3. wheter an edge -> whether an edge.
Thank you for your review and your helpful suggestions.
1) On the effect of sample size: we agree that the sample size directly affects the performance of these methods. We investigated this effect in Appendix D.2 (which is now Appendix E.4 in the revision), where we ran the same experiment on Wikipedia English with batch sizes 128, 512 (Tables 3 and 4), and compared the results to batch size 1024 (Table 2). We simultaneously varied the learning rate to understand its effect as well, but focusing on the effect of batch size only, we can observe that
(i) the performance of all methods increases with the batch size (at least in the 128-1024 range).
(ii) the relative improvement of our methods (compared to the baseline) is larger for smaller batch sizes: the relative improvement is 19.5% for 1024, 26.7% for 512, and 29.1% for 128.
Of course, one cannot increase the batch size indefinitely as there are hard limits on memory size, and the key advantage of our methods is in problems where sampling-based methods give poor estimates even with the largest feasible batch size.
The effect of the batch size can also be seen to some extent in Figure 2.a, where we show the quality of the Gramian estimates for batch size 128 and 1024. The figure suggests that the quality improves, for all methods, with larger batch sizes, and that SOGram with batch size 128 has a comparable estimation quality to the baseline with batch size 1024.
2) The reviewer raises an interesting point. We have observed in our experiments that for a fixed sampling distribution, improving the Gramian estimates generally leads to better MAP, but we cannot draw conclusions when the sampling distribution changes. One possible explanation is that the sampling distribution affects both the quality of the Gramian estimates, and the frequency at which the item embeddings are updated.
In particular, tail items are sampled more often under uniform sampling than under the other distributions, and updating their embeddings more frequently may contribute to improving the MAP. We added a comment (Appendix E.2 in the revision) to highlight this observation.
Summary of the paper:
This work presents a novel method for similarity function learning using non-linear model. The main problem with the similarity function learning models is the pairwise component of the loss function which grows quadratically with the training set. The existing stochastic approximations which are agnostic to training set size have high variance and this in-turn results in poor convergence and generalisation. This paper presents a new stochastic approximation of the pairwise loss with reduced variance. This is achieved by exploiting the dot-product structure of the least-squares loss and is computationally efficient provided the embedding dimensions are small. The core idea is to rewrite the least-squares as the matrix dot product of two PSD matrices (Grammian).
The Grammian matrix is the sum of the outer-product of embeddings along the training samples. The authors present two algorithms for training the model, 1)SAGram: By maintaining a cache of all embedding vectors of training points (O(nk) space)$, whenever a point is encountered it's cache is replaced with it's embedding vector. 2) SOGram: This algorithm keeps a moving average of the Grammian estimate to reduce the variance. Experimental results shows that this approach reduces the variance in the Grammian estimates, results in faster convergence and better generalisation.
Review:
The paper is well written with clear contribution to the problem of similarity  learning.  My only complain is that, I think the evaluation is a bit weak and does not support the claim that is applicable all kinds of problems e.g. nlp and recommender systems. This task in Wikipedia does not seem to be standard (kind of arbitrary) — there are some recommendation results in the appendix but I think it should have been in the main paper.
Overall interesting but I would recommend evaluating in standard similarity learning for nlp and other tasks (perhaps more than one)
There are specific similarity evaluation sets for word embeddings. It can be found in following papers: https://arxiv.org/pdf/1301.3781.pdf
http://www.aclweb.org/anthology/D15-1036
Thank you for your assessment and your helpful suggestions.
Regarding evaluation: since the focus of the paper is on the design of an efficient optimization method, we wanted to choose an experiment where (i) the evaluation metric is aligned with the optimization objective, and (ii) the vocabulary size is very large (on the order of 10^6 or more), making traditional sampling-based methods inefficient, because they would require too many samples to achieve high model quality. This is why we chose the Wikipedia dataset, which is, to our knowledge, one of the few publicly available datasets of this scale. It also offers different subsets of varying scale, which allowed us to illustrate the effect of the problem size, suggesting that the benefit of the Gramian-based methods increases with vocabulary size. We added a note to the revision to comment on our choice.
We also agree that it will be beneficial to evaluate these method on other applications such as more traditional natural language tasks, and this is something we intend to pursue in future work.
We would like to thank all reviewers for their careful reading and helpful suggestions. We have uploaded a revision of the paper with the following changes:
- We added a new section to the appendix (Appendix C) discussing how to adapt the methods to a non-uniform weight matrix.
- We added Appendix E.1 to relate the gradient estimation error to the Gramian estimation error, with a numerical experiment (Figure 6) showing the effect of our methods on gradient estimates.
- We added a comment to the conclusion to emphasize that our experiments were focused on problems with very large vocabulary size.
- We rearranged the introduction, and improved transitions between sections.
- We added comments to the numerical experiments (Section 4 and Appendix E) highlighting the effect of the batch size and of the sampling distribution.
We thank the reviewers again for their time and helpful comments.

Meta-review:
role: AC
data: metareview: This paper presents methods to scale learning of embedding models estimated using neural networks. The main idea is to work with Gram matrices whose sizes depend on the length of the embedding. Building upon existing works like SAG algorithm, the paper proposes two new stochastic methods for learning using stochastic estimates of Gram matrices. 

Reviewers find the paper interesting and useful, although have given many suggestions to improve the presentation and experiments. For this reason, I recommend to accept this paper.

A small note: SAG algorithm was originally proposed in 2013. The paper only cites the 2017 version. Please include the 2013 version as well.


Decision: Accept (Poster)