Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: There are many ways to reduce the memory footprint and increase speed of a neural network: weight quantisation, compression, coarse-to-fine, knowledge distillation, etc. The method proposed in this work is a specific case of knowledge distillation that focuses on the discrete-input-to-first-layer and output-layer-to-discrete-output transformations, which represent a large portion of the parameters.

The authors propose to use a variant of SVD (which can be viewed as 2 linear transformation, with a middle dimension that represents an embedding), where the first transformation is linear with a ReLu, and the second is linear. By approximating the learned matrices of the model, the experiments show that using the proposed variant of SVD gives similar predictive performance compared to the original model, with a fraction of the parameters.

However, it seems that the authors could have simply replaced the input by a 2-layer NN (first a linear+ReLu, then a Linear) to obtain the same parametrisation, but they could have learned the parameters in a end-to-end fashion. It is not clear to me why using a surrogate L2 loss within the model should give better predictive performance than a fully end-to-end trained neural network. Without this comparison, I do not think the proposed experiments are conclusive enough.


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes to use low-rank matrix decomposition for embedding compression, with relu in the reconstruction layer to gain non-linearity. Experiments on machine translation task shows improvement compared with state-of-the-art methods with different compression rates.

Detailed comments:
1)	The technical contribution seems to be a bit limited. Using relu in the reconstruction function looks straightforward and adding reconstruction loss in objective function is also common practice. Also, not much insight is provided on why such approach works better than other baselines. 

2)	Experiments:
a.	It is good to see such simple approach outperforms several more sophisticated baseline methods. Also, ablation study is also performed to show the effect of different components.

b.	How does the time complexity and running time of the proposed method compared to the baselines?

c.	The paper only evaluates distilled embedding on one task (i.e., machine translation). The experiments would be more convincing if evaluated on more tasks as well.

d.	It could be helpful to include some sensitivity analysis on the hyperparameters such as \alpha which controls the weight of reconstruction loss. 

In conclusion, this paper seems to be below the bar and I would recommend a ‘weak reject’ for the paper.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: 

This paper proposes a method for compressing embedding matrices of both encoder/decoder embeddings.
The basic idea of the proposed method is to reconstruct the embedding matrix by what they called the “funneling decomposition” method, whose parameter shape is identical to the SVD (low-rank matrix) decomposition with additional non-linear function.
Therefore, the idea itself is not so novel and innovative.
Moreover, their method requires the embedding matrix as the teacher signal for calculating the reconstruction loss.
We need to note that the memory requirement of the proposed method during training will increase.
 
One of the notable advantages of the proposed method is that their proposed method seems to successfully reduce the embedding matrix even if it shares the parameters with the output layer, which is a de-facto standard model architecture for NMT.
As pointed out by the authors, this seems to be the first success of reducing the embedding matrix with a tied embedding setting.


1,
The authors claim that “We demonstrate that at the same compression rate our method outperforms existing state-of-the-art methods.” at the end of the Introduction section.
However, according to Tables 1, 2, and 3, it seems that the performance gain is marginal compared with similar methods.
For example, 
37.78 (proposed) <=> 37.78 (Shi & Yu (2018)) 
26.97 (proposed) <=> 26.75 (Chen et al. (2018)
and
 42.62 (proposed) <=> 42.37 (SVD with rank 64),
which are the at most 0.25 BLEU gain.
I believe that most of MT researchers hardly say that BLEU 0.25 difference is a significant improvement. Besides, the authors should perform a statistically significant test if they say “our method outperforms existing state-of-the-art methods.”
 
 
2
I am a bit confused about the following inconsistency;
The authors say that “Compressing embedding matrices without sacrificing model performance is essential for successful commercial edge deployment” in the abstract.
However, according to Table 1, the number of parameters for embeddings is 16.3M, which is only 27% of the total number of parameters in Transformer base.
By this fact, compressing embedding matrices seems not essential for successful commercial edge deployment.
 
In Table 6, it is explicitly unclear what is the difference between 
“Funneling with Emb. Distillation”, “Funneling (with non-linearity),” and “Funneling (with retraining all weights).”
Please give us a more precise explanation.

review_title: Official Blind Review #2

Meta-review: This paper proposes to further distill token embeddings via what is effectively a simple autoencoder with a ReLU activation. All reviewers expressed concerns with the degree of technical contribution of this paper. As Reviewer 3 identifies, there are simple variants (e.g. end-to-end training with the factorized model) and there is no clear intuition for why the proposed method should outperform its variants as well as the other baselines (as noted by Reviewer 1). Reviewer 2 further expresses concerns about the merits of the propose approach over existing approaches, given the apparently small effect size of the improvement (let alone the possibility that the improvement may not in fact be statistically significant).

Decision: Reject