Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a general embedding method, Anchor & Transform (ANT), that learns sparse representations of discrete objects by jointly learning a small set of anchor embeddings and a sparse transformation from anchor objects to all the others.

Strengths of the paper:
1. The paper is well-written and easy to be followed.
2. The research problem is of great value to be investigated.

Weaknesses of the paper:
1. The idea of utilizing anchors to reduce the size of features (in your case, the total embeddings of discrete objects to be inferred) has been widely studied in related fields in computer science. For instance, there are a number of papers in the field of manifold learning using anchors to reduce the size. The inherent connections and relationships between the proposed methods and other algorithms using anchors should be carefully discussed.
2. The contribution of the paper seems not significant, as the idea of utilizing anchors to reduce the number of parameters to be inferred has been widely studies in the related work. There are a number of papers utilizing anchors, such as the followings (just list some of them):
B. Xu et al., Efficient manifold ranking for image retrieval, in SIGIR 2011.
S. Liang et al., Manifold learning for rank aggregation, in WWW 2018.
Y. Guo et al., Learning to hash with optimized anchor embedding for scalable retrieval, in TIP, 2017.
In the last reference aforementioned above, both anchors and embeddings are jointly taken into account. 
3. Baselines should include some manifold learning algorithms that take anchors into account.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This manuscript proposed to represent the embedding matrix as a small set of anchor embedding and sparse transformation. The paper is trying to be general-purpose, end-to-end trainable, and able to incorporate domain knowledge. Experimental results show that it is possible to compress the embedding in the proposed way without much loss of accuracy.
 
The authors propose to find anchor embedding by several methods, such as frequency, clustering, or random sampling. The sparsity on the transform is imposed by L_1. Although I get the basic idea and I am familiar with many of the techniques, it is unclear to me what is the main focus of this paper, and the technical contribution is quite vague. Why is the large embedding matrix a problem? Besides the low-rank form proposed, are there any other ways to compress it? This paper is not well motivated at all. Therefore, I think this manuscript is not ready to publish in its current form.

#####
Thank you for the response! I've increased my score to 3: Weak Reject. Although the idea of compressing the (word) embedding layer using low-rank structures is not new (even with the end-to-end training), the main technical contribution in this paper is to jointly learn the anchor embedding (anchor pre-selected with multiple schemes) and sparse transformation (sparsity achieved via Proximal GD). Moreover, domain knowledge can be incorporated by adding specialized constraints such as orthogonality and selective penalization. 

At first glance, the idea presented in this paper seems not new, and I doubt many people are doing similar stuffs already in practice. I find the explanations on the technical points in Appendix C helpful. The empirical study in this paper looks strong. The authors considered experiments in text classification and language modeling with a number of baselines, which demonstrates the advantages of anchor and joint training in the proposed way. This paper presents several useful heuristics around, but I share the concern with other reviewers about whether the main point is compelling enough, given the existing body of work along with this line. 


  


  
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper describes a "layer" that aims at producing embeddings for discrete objects by using fewer parameters than classical embeddings layers. Indeed, the model proposes, instead of learning an embedding matrix of size VxN, to learn a matrix of embeddings of anchors (AxN) and a transformation matrix (VxA) such that the embedding of any object can be found by multiplying A with T. On top of that, they propose different regularization techniques to improve the quality of the learned embeddings, and particularly a proximal gradient method over a L1 normalization on T to reduce the number of parameters. They propose also different ways to initialize A and also a method for incorporating a priori information (e.g knowledge) into the model.  They evaluate this model on different tasks: text classification and language modeling and show that they can achieve good performance while using fewer parameters than Sota methods. 

First of all, the paper is well written, and the description is very detailed and understandable. It was a pleasure to read such a paper!  

One point which is unclear is the interest of using such a method, and more precisely in which cases, this method can be useful. Indeed, the overall number of parameters of ANT is AxN + VxA (N being the size of the embeddings, A the number of anchors and V the size of the vocabulary) while classical methods are VxN parameters. Said otherwise, we need to have V<N to really have less parameters to train in the model -- knowing that classical embeddings spaces size is usually between 256 and 1024, it means that we have to target a task where the number of anchors is quite low. I agree that the sparsity term on T is here to encourage to decrease the number of parameters but first, the same sparsity could be applied on the original VxN embedding matrix, and also, even if, at the end, the T matrix is sparse, during learning one has to maintain a large matrix in memory.  I would like the authors to discuss more on this point which is crucial? Particularly, I am not sure to understand what the #Emb value is in the table (AxN + AxV or just AxN), and how to compare the models. (There is a discussion in Section 3, but the argumentation does not explain why having so many parameters at train time is not a problem).  Also, since this is the crucial point in the paper, I would be interested in having a discussion about the use of neural models compression techniques after learning that could also "do the job" (even if they are not trained end-to-end). 

One other remark concerns the different "components" added into the model (e.g sparsity, orthogonality, Relu...). It is difficult to measure the interest of each of them, and I would recommend the authors to provide an ablation study to make the effect of the different choices more understandable by the reader.

The notion of anchors also is misleading since it gives the impression that the A matrix will store embeddings for particular objects, while there is no constraint of that type. Each line of the A matrix is an embedding, but this embedding is not associated with one of the objects seen at train time (no direct mapping from anchors to words in the vocabulary). This has to be made more clear at the beginning of the paper. 

Concerning the initialization of A by K-means, it assumes that the space of objects has a particular metric. The authors say that this metric can come from a pretrained embedding space, but in that case, the problem in the number of parameters (which is the main justification of this work) is invalid (i.e if you already have an embedding matrix, then just let us fine-tune it). Could you clarify ? 

The fact that the method would allow incorporating knowledge is certainly the most interesting point. The way it is done has to be better explained (I do not understand why positive pairs are taken into account by not enforcing sparsity on T at this particular point, the way negative pairs are handled seem more natural)

The paper is interesting and proposes a new simple model that could be used to keep good performance while reducing the number of parameters of the final model. Discussions have to be added to discuss the relevance of the approach since it still needs a large number of parameters at train time, and the role of each component could be studied more in depth. 
review_title: Official Blind Review #3

Meta-review: The paper proposes a method to produce embeddings of discrete objects, jointly learning a small set of anchor embeddings and a sparse transformation from anchor objects to all the others. While the paper is well written, and proposes an interesting solution, the contribution seems rather incremental (as noted by several reviewers), considering the existing literature in the area.  Also, after discussions the usefulness of the method remains a bit unclear - it seems some engineering (related to sparse operations) is still required to validate the viability of the approach.

Decision: Reject