Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
This paper explores two related methods to reduce the number of parameters required (and hence the memory footprint) of neural NLP models that would otherwise use a large word embedding matrix. Their method, inspired by quantum entanglement, involves computing word embeddings on-the-fly (or by directly computing the output of the "word embedding" with the first linear layer of network). They demonstrate their method can save an impressive amount of memory and does not exhibit big performance losses on three nlp tasks that they explore.

This paper is clearly written (with only a couple of typos) but does not yet reach publication standard. Whilst the empirical performance of their approach is promising from the perspective of saving reducing memory requirements, more experiments are required and more careful comparisons to baselines and other methods in the literature for saving memory/parameters. In general the related work and experimental sections are weak and brief, with only superficial analysis. There is  lack of careful analysis and insight into their results, as well as a careful comparisons to other work in this area.

The choice of tasks to evaluate on is broad, which is a strength, but is missing simpler tasks that one would expect to see, such as a text classification dataset, or simple bag-of-vectors style models. In addition, the choice of models are somewhat outdated baselines. It seems that transformers would be an ideal setting for their approach, as transformers have rather high dimensional word embedding matrices, but the authors do not run experiments with the now-ubiquitous Transformer.

The quantum inspiration is largely a distraction, and I think the paper would benefit from this element being scaled back or removed in order to free up space for more experiments.

The authors acknowledge one key weakness of their approach, that both training and inference time are increased (by 28% or 55% longer for DocQA depending on compression)  but much more work could be done to understand the best way to  mitigate for longer training and inference times.

The authors argue that reducing the memory footprint of models is vital to address hardware limitations for training and inference for large models like BERT or ROBERTA, but this argument is not particularly strong. Generally current limitations for training these kinds of models  are the long training times and being able to fit large batches onto our hardware, and the vocabulary matrix is only a constant factor here. And since training time is a bottleneck, the added value of saving memory vs slowing the training speed by 30-50% is debatable. 

Here are some questions for the authors that come to mind when reviewing:

How does your method compare to other published methods on your benchmarks? 

which choices for r and k lead to the best time/memory/performance tradeoff? how does this compare to other compression methods (on your tasks)

Seq2Seq models usually involve multiplying the the output hidden state with a vocab matrix before softmaxing over all the vocabulary produce word probabilities - did you account for this? Does your method work for the output vocab matrix? 

Did you investigate pre-training word2ket like word2vec or Glove?

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: 
This paper proposes word2ket - a space-efficient form of storing word embeddings through tensor products. The idea is to factorize each d-dimensional vector into a tensor product of much smaller vectors (either with or without linear operators). While this results in a time cost for each word lookup, the space savings are enormous and can potentially impact several applications where the vocabulary size is too large to fit into processor memory (CPU or GPU). The experimental evaluation is done on several tasks like summarization, machine translation and question answering and convincingly demonstrates that one can achieve close to original model performance with very few parameters! 

This approach would be very useful due to growing model sizes in many areas of NLP (e.g. large pre-trained models) and more broadly, deep learning.

Pros:
1. Novel idea, clear explanation of the method and the tensor factorization scheme. 
2. Convincing experiments on a variety of NLP tasks that utilize word embeddings. 

Cons:
1. (Minor) While this is not the focus of the paper, it would be useful to have at least one experiment with a state-of-the-art model on any of these tasks to further strengthen the results (most of the baseline models used currently seem to be below SOTA).


Minor comments:
Abstract: stain -> strain
Page 2: $||u|| \rightarrow ||w||$
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper presents two methods to learn word embedding matrices that can be stored in much less space compared to traditional d x p embedding matrices, where d is the vocabulary size and p is the embedding size. Two methods are proposed: the first method estimates a p-dimensional embedding for a word as a sum of r tensor products of order n (tensor product of n q-dimensional embeddings).  This representation takes rnq parameters which can be much less than p, since p = q^n. The second method factorizes a full d x p embedding matrix jointly as a tensor product of much smaller t x q matrices and can obtain even larger space savings. Algorithms for efficiently computing full p-dimensional representations are also included. When only dot products are needed, the p-dimensional representations do not need to be explicitly constructed.

In my opinion the terminology from quantum computing and entanglement is an unnecessary complication. It would be better to simply talk about the special parametric form of the embeddings , which allows efficient storage. Tensor product representations have been used for embeddings before (but not with the goal of efficiency) (e.g. Arora et al 2018) https://openreview.net/pdf?id=B1e5ef-C-
The paper covers related work briefly and does not compare experimentally to any other work aiming to reduce memory usage for embedding models (e.g. using up-projection from lower-dimensional embeddings, or e.g. this paper: Learning Compact Neural Word Embeddings by Parameter Space Sharing by Suzuki and Nagata.

The experimental results on summarization, machine translation, and QA show that the methods can obtain comparable results to models using traditional word embeddings while obtaining savings of up to one-thousand fold decrease in space needed for the embeddings.

The experimental results seem to conflate the issues of the dimensionality of the word embeddings versus that of the higher layers. For example, in the summarization experiments, word2ketXS embeddings corresponding to 8000-dimensional embeddings are compared to a standard model with embeddings of size 256. The LSTM and layers for the word2ketXS model would become quite large but their size is not taken into account. In addition, the activation memory is often the major bottleneck and not the parameter memory. These issues are not discussed or made explicit in the experiments.

Overall the paper can be a strong contribution if the methods are stated with less quantum computing jargon, the overall parameter size and speed of the different models is specified in the experiments, and more specific connections to related work are made. Ideally, an experimental comparison to a prior method for space-efficient embeddings.

Question: What is the role of pre-trained Glove embeddings in the word2ket models? Was any pre-training done on unlabeled text?


Some typos:

Section 1.1

“matrix, as the cost ..”  -> “matrix, at the cost”

Under Eq (2)
I think you mean w instead of u 

Section 3.2

F_j: R^t -> R^p , do you mean R^q 

review_title: Official Blind Review #1

Meta-review: This paper proposes quantum-inspired methods for increasing the parametric efficiency of word embeddings. While a little heavy in terms of quantum jargon, and perhaps a little ignorant of loosely related work in this sub-field (e.g. see the work of Coecke and colleagues from 2008 onwards), the majority of reviewers were broadly convinced the work and results were of sufficient merit to be published.
Decision: Accept (Spotlight)