Distilled embedding: non-linear embedding factorization using knowledge distillation
Authors: Vasileios Lioutas
Ahmad Rashid
Krtin Kumar
Md Akmal Haidar
Mehdi Rezagholizadeh
Venue:  (2020)
Keywords: Model Compression
Embedding Compression
Low Rank Approximation
Machine Translation
Natural Language Processing
Deep Learning

Abstract: Word-embeddings are a vital component of Natural Language Processing (NLP) systems and have been extensively researched. Better representations of words have come at the cost of huge memory footprints, which has made deploying NLP models on edge-devices challenging due to memory limitations. Compressing embedding matrices without sacrificing model performance is essential for successful commercial edge deployment. In this paper, we propose Distilled Embedding, an (input/output) embedding compression method based on low-rank matrix decomposition with an added non-linearity. First, we initialize the weights of our decomposition by learning to reconstruct the full word-embedding and then fine-tune on the downstream task employing knowledge distillation on the factorized embedding. We conduct extensive experimentation with various compression rates on machine translation, using different data-sets with a shared word-embedding matrix for both embedding and vocabulary projection matrices. We show that the proposed technique outperforms conventional low-rank matrix factorization, and other recently proposed word-embedding matrix compression methods. 


Under review as a conference paper at ICLR 2020
DISTILLED EMBEDDING: NON-LINEAR EMBEDDING FACTORIZATION USING KNOWLEDGE DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Word-embeddings are vital component of Natural Language Processing (NLP) systems and have been extensively researched. Better representations of words have come the cost of huge memory footprints, which has made deploying NLP models on edge-devices challenging due to memory limitations. Compressing em- bedding matrices without sacrificing model performance is essential for successful commercial edge deployment. In this paper, we propose Distilled Embedding, an (input/oupput embedding compression method hased on low-rank matrix decom- position with an added non-linearity. First, we initialize the weights of our decom- position by learning to reconstruct the full word-embedding and then fine-tune on the downstream task employing knowledge distillation the factorized embed- ding. We conduct extensive experimentation with various compression rates on machine translation, using different data-sets with a shared word-embedding ma- trix for both embedding and vocabulary projection matrices. We show that the proposed technique outperforms conventional low-rank matrix factorization, and other recently proposed word word-embedding matrix compression methods.
INTRODUCTION
Deep Learning models are the state-of-the-ar in NLP, Computer Vision, Speech Recognition and many other fields in Computer Science and Engineering. The remarkable deep learning revolution has been built on top of massive amounts of data (both labeled and unlabeled), and faster computa-
tion. In NLP, large pre-trained language models like BERT (Devlim al. 2018) are state-of-the-arr on a large number of downstream NLP problems. The largest publicly available language model to date is trained with 1.6 billion parameters (Keskar et al., 2019). On machine translation the state- of-the-art models have parameters in the order of millions. Data privacy and server cost are some major issues, driving research towards deploying these models on edge-devices. However, running these models on edge-devices, face memory and latency issues due to limitations of the hardware. Thus, there has been considerable interest towards research in reducing the memory footprint and faster inference speed for these models (Sainath et al. 2013 Acharya al. 2019 Shu & Nakayama
2017 Shi & Yu 2018; Jegou et al. 2010 Chen et al. 2018 Winata et al.
2018
The architecture of deep-learming-basse based NLP models can be broken down into three components. The first component, represents the embedding section, which maps words in the vocabulary to con- tinuous dense vector representations of the words. For all future references, the first component includes both the source and target vocabulary mapping using a common embedding matrix with a shared vocabulary. The second component, consists of function f. typically deep neural-network [Schmidhuber 2015 Krizhevsky et al. 2012 Mikolov et al., 2010) which maps the embedding rep- translation, summarization, question-ange answering and others), to the output-space of function f. The third component, is the output layer which maps the output of function f the vocabulary-space followed by softmax function. Most of these NLP problems use linear transformations, in the first and the third components of the models. Since, these components depend upon large vocabulary-size they require large number of parameters which results in higher latency and larger memory requirements. For instance, the Transformer Base model (Vaswani et al. 2017) uses 37% of the parameters in the first and third components using a vocabu- lary size of 50k, and with purameter-tying between the components. The percentage of parameters increases to 54%, when parameters are not shared between the first and third components. The reader is encouraged to look at Table A. the Appendix for more information.
resentation for different NLP problems

---

Under review as a conference paper at ICLR 2020
Thus, obvious step is to compress parametric functions used by the first and third components. Recently, many researchers have worked on compressing embeeddin matrices (Sainath et al.
2013
Acharya al. 2019 Shu & Nakayama 2017 Shi & Yu 2018 Jegou et al. |210|] Chen al.
2018
Winata et al. 2018). These techniques have proven to perform at-par with the uncompressed
models, but still sufter from number of issues.
First, embedding compression models (Shi & Yu 2018 Chen et al. 2018 Khrulkov et al. 2019
Shu & Nakayama 2017), require two hyper-parameters
be fine-tuned. These hyper-parameters influence the number of parameters in the model, and thus the compression rate. This leads to an additional layer of complexity for optimizing the model for different NLP problems. Additionally,
Chen et al. (2018) 8) requires an additional optimization step for grouping words, and lacks end-to-end training through paack-rrpagationn [hi & Yu (2018) also requires an additional step for performing k-means clustering for generating the quantization matrix. Thus, most of the current state-of-the- systems are much more complicated fine-tune for different NLP problems and data-sets,
Second, all the state-of-the-art embedding compression models compress the first and third compo- nents separately. In practice, state-of-the NLP models like Vaswani et al. (2017) have shown to perform better with parameter sharing between the first and third components (Press& & Woll 2016). Thus, there is a need for an exhaustive analysis of various embedding compression techniques, with parameter sharing.
Lastly, embedding compression models not based on linear SVD (Khrulkov et al. (2019); Shu & Nakayama (2017); ; (Shi& & Yu (2018)) require the reconstruction of the entire embedding matrix or additional computations, when used at the output-layer. Thus, the model either uses the same amount of memory as the uncompressed model or requires additional computation cost. This makes linear SVD based techniques more desirable for running models on edge-devices.
In this paper, introduce Distilled Embedding, non-linear matrix factorization method, which outperforms the current statc-of-the-ar methods. Our method, first compresses the vocabulary-space much smaller dimension compared the original hidden--imension, then applies non-linear activation function, before recovering the original introduce an embedding distillation method, which similar to Knowledge Distillation (Hinton et al., 2015) but we apply it to distill knowledge from a pre-trained embedding matrix and use an L2 loss instead of cross-entropy loss. summarize our contributions are:
-imdddiin-dimmsiin. Additionally, we also
We demonstrate that at the same compression rate our method outperforms existing state- of- of-the-an art methods.
Our proposed method is much simpler than the current state-of-the-method with only single hyper-parameter controlling the compression rate.
We introduce embedding distillation method, which out-performs standard machine translation cross-entropy loss.
Unlike the current oftte-of-the-art systems, we compress the embedding matrix with param- eter sharing between the first and third components. We perform an exhaustive comparison of current state-of-the-ee -ort models in this setting.
Our model uses similar factorization SVD, with the addition of non-linearity, thus it is able to leverage benefits of linear SVD based methods discussed above.
We also compare all the models against simple low-rank matrix factorization using Sin- gular Value Decomposition (SVD) and demonstrate that it can compete with more complex algorithms.
2 RELATED WORK
We can model the problem of compressing the embedding matrix as a matrix factorization prob- lem. There is considerable amount of work done in this field, some of the popular works broadly belong to the domain of low-rank factorization (Singular Value Decomposition (SVD) Srebro & Jaakkola (003)): Mnih Salakhutdinov (2008), product quantization (Jegou et al. 2010) and tensor decomposition Dee Lathauwer et al.) 2000). A number of prior works in embedding compression are influenced by these fields and have been applied to various NLP problems. In this Section, we will discuss some of the significant works across different NLP problems.
2

---

Under review as a conference paper at ICLR 2020
Low-rank Factorization Low-rank approximation of weight matrices, using SVD, is natural way to compress deep learning based NLP models. [Sainath et al. (2013) apply this to a convolutional neural network for language modeling and acoustic modeling. Winata et al. (2018) use SVD on all the weight matrices of an LSTM and demonstrate competitive results on question-answering, language modeling and text-entailment. (Acharya al. (2019) use low-rank matrix factorization for word-embedding layer during training to compress a classification model. However, they do not study the effects of applying non-linear function before reconstructing the original dimension.
GroupReduce Chen et al. 2018) apply weighted low-rank approximation to the embedding ma- trix of an LSTT. They first create a many-to-one mapping of all the words in the vocabulary into groups, this initial mapping is done based upon frequency. For cach group they apply weighted SVD to obtain lower rank estimation, the rank is determined by setting the minimum rank and linearly increasing it based upon average frequency. Finally, they update the groups by minimizing the reconstruction error from the weighted SVD approximation. They demonstrate strong results on language modeling and machine translation compared to simple SVD. In their models they use different embedding matrices for input and softmax layers and apply different compression to each.
Product Quantization Jegou et al. 2010) introduced product quantization for compressing high dimensional vectors, by uniformly partitioning them into subvectors and quantizing each subvector using K-means clustering technique. Basically, product quantization approach assumes that the subvectors share some underlying properties which can be used to group similar subvectors together and unify their representation. That being said, this approach breaks the original matrix into set of codebooks coming from the center of the clusters in different partitions together with a separate index matrix which refers to the index of the clusters for each subvector. [Shi & Yu (2018) applied product quantization to a language model and were able to show better perplexity scores. [hh & matrix factorization problem, and then learning the quantization matrix in an end-to-end trainable neural network. Li et al. (2018) implement product quantization through randomly sharing parameters in the embedding matrix, and show good results on perplexity for an LSTM based language model.
Nakayama (2017) extended this technique by first representing the product quantization
(2000) introduced multilinear SVD, which is a gener- alization of SVD for higher order tensors. Oseledets (201T) introduced an efficient algorithm Tensor Train (TT) for multilinear SVD Tensor. Novikov et al. (2015) applied the Tensor Train decompo- sition on fully connected layers of deep neural networks. Khrulkov et al. (2019) applied Tensor Train algorithm to the input embedding layer on different NLP problems like language modeling, machine translation and sentiment analysis. They demonstrate high compression rate with little loss of performance. However, they compress only the input embedding and not the softmax layer for language modeling and machine translation.
Tensor Decomposition De Lathauwer
Knowledge Distillation Knowledge distillation has been studied in model compression where knowledge of large cumbersome model is transferred to small model for easy deployment. Sev- cral studies have been studied on the knowledge transfer technique (Hinton et al. (2015); Romero et al. 2)15)). In this paper, we propose embedding factorization of word-embedding matrix using knowledge distillation to mimic the pre-trained word-embedding representation.
METHODOLOGY: DISTILLED EMBEDDING
3.1 FUNNELING DECOMPOSITION AND EMBEDDING DISTILLATION
We present an overview of our proposed method in Figure Given an embedding matrix E e R V xd we can decompose it into three matrices (Equation 1b. using the SVD algorithm
E = UpP E V
(1)
where | is the vocabulary size and the embedding dimension. Σ is a diagonal matrix con- taining the singular values, and matrices U and V represent the left and right singular vectors of the embedding matrix respectively. We can obtain the reduced form of the embedding matrix by only keeping r (< d) largest singular values out of d.
F Xp Exxr V Upiir
V.T. Vixaa
(2)
3

---

Under review as a conference paper at ICLR 2020
Lrecon
Liotal
Toron
(1 αx) Co
Decomposes
V
(Transposed)
Neural Decimposition
14
ReLU
Bottleneck Size
Tharwformer
Encoder
Decoder
U
Shared Decomposed Empesdingg
Shared Decomposed Embedding
Inputs
Outputs (shifued right)
Pre-trained
Embedding
M
(b) Finetuning all the weights with knowledge distilla- tion the embedding
(a) Funneling Decomposition
Funneling Decomposition method to compress the shared embedding matrix of trans- former based sequence to sequence model.
Figure 1:
where the matrix U = UE. The reduced form of the embedding matrix will need r x ((V) + parameters compared to |V] x d.
Our proposed approach in this work, is to apply non-linear transformation on the matrix U, before reconstructing the original embedding dimension using (see Figure la), as shown Equation
We use the ReLU as our non-linear function f(.) throughout this paper. We train a sequence to sequence model (Sutskever et al. (2014); Vaswani et al. (2017) with tied input and output embed- ding (i.e. the output embedding the transpose of the input embedding matrix Exu ET
Vaxr [U We train our model end-to-end by replacing the embedding function with Equa- tion 3 The matrix and V trainable parameters, and for the output layer we use ET, with the parameter sharing. We train on two losses. The standard cross entropy loss defined as:
M
Lee =
Σ y, log(p:)
(4)
where M is the sequence length, y; the one-hot representation for the 7th label and p. the softmax probability of the 7th term generated by the decoder.
In addition to the cross-entropy loss, we introduce a novel embedding reconstruction loss (Equa- tion (5). which we refer to embedding distillation as we distill information from the pre-trained embedding into our model,
V
M
1
Crecon
lle E112
Σ lle
f(u))
V
where and é; are the embedding vectors corresponding to the th word in the original embedding matrix E and the reconstructed embedding matrix á respectively and u, refers to the 7th row of the matrix U. We use Equation 6 as our final loss function.
Lioal = ChCreson + (1 - Loe
(6)
where [0, 1] is hyper-paramette which controls the trade-off between reconstruction and cross-entropy loss. Creom acts as the knowledge distillation loss by which we try to distill informa- tion from the original pre-trained embedding layer teacher to the funneling decomposed embed- ding layer as a student. The training process our Distilled Embedding method is summarized in Algorithm
4

---

Under review as a conference paper at ICLR 2020
Algorithm Distilled Embedding
Step 1) Pre-training the Embedding Matrix Pre-train the sequence to sequence model with the full embedding matrix for better initialization.
Step 2) Initializing the Weights of Funneling Decomposition Layer We extract the trained embedding matrix E' from Step 1 and train our decomposed matrices U and V on reconstruction loss defined in [ as shown in Figure Ial
Step 3) Embedding Distillation The pre-trained funneling decomposition layer is plugged into the model (replacing the original embedding matrix E) and the entire model is trained based on Equation 6
EXPERIMENTAL SETUP
4.1 DATASETS AND EVALUATION
We test our proposed method on machine translation which is a fundamental problem in NLP and challenging for embedding compression since we typically have at least two dictionaires and an input and output embedding. Comparitively language modeling uses single dictionary and classification tasks such as sentiment analysis don't 't have an output embedding.
We present results on translating three language pairs: WMT English to French (En-Fr), WMT English to German (En-De) and IWSLT Portuguese to English (Pt-En). We decided that these pairs are good representatives of high-resource, medium-resoure, and low-resource language pairs.
WMT En-Fr is based on WMT14 14 training data which contain 36M sentence pairs. We used Senten- cePiece (uudo Richardson 2018)) to extract shared vocabulary of 32k subwords. We validate on newstest2013 and test on newstest2014, 4. For WMT English to German (En-De), we use the same setup as Vaswani et al. (2017). The dataset is based on WMT16 training data and contains about 4.5M pairs. We use a shared vocabulary of 37k subwords extracted using SentencePiece.
For the IWSLT Portuguese to English (Pt-En) dataset, we replicate the setup of Tan et al. (2019) for training individual models. Specifically, the dataset contains about 167k training pairs. We used shared vocabulary of 32k subwords extracted with SentencePiece.
For all language pairs, we measure case-sensitive BLEU score (Papineni et al. (2002) using Sacre- BLEU (Post 2018)). In addition, we save checkpoint every hour for the WMT En-Fr and WMT En-De language pairs and every minutes for the IWSLT Pt-En due the smaller size of the dataset. We use the last checkpoint which resulted in the highest validation BLEU and average the last five checkpoints based on this. We use beam search with beam width of 4 for all language pairs.
4.2 EXPERIMENT DETAILS
Sypee-Parraeters For WMT En-Fr and WMT En-De we use the same configuration as Trans- former Base which was proposed by Vaswani et al. (0177). Specifically, the model hidden size dmodel is set to 512, the feed-forward hidden size dr is set to 2048 and the number of layers for the encoder and the decoder was set 6. For the IWSLT Pt-En, we use Transformer Small configu- ration. Specifically, the model hidden-size dmotel is set to 256, the feed-forward hidden size di is set to 1024 and the number of layers for the encoder and the decoder was set to 2. For Transformer Small, the dropout configuration was set the same Transformer Base. All models are optimized using Adam (Kingma & Bal (2015) and the same learning rate schedule as proposed by Vassann et al. 2017). We use label smoothing with 0.1 weight for the uniform prior distribution over the cabulary (Szegedy et al. (2015): Pereyra et al. (2017)). Additionally, we set the value a of Equation to 0.01.
Hardware Details We train the WMT models on 8 NVIDIA V100 GPUs and the IWSLT models on single NVIDIA V100 GPU. Each training batch contained set of sentence pairs containing
5

---

Under review as a conference paper at ICLR 2020
Emb. Params
Emb.
WMT En-Fr
Model
Param
Compression Rate
(BLEU)
Transformer Base
60M
16.3M
1.0x
38.12
Smaller Transformer Network (416)
46M
13.3M
1.23x
37.26
SVD with rank 64
46M
2.08M
7.87x
37.44
End-to-End NN with non-linearity
46M
2.08M
7.87x
37.23
GroupReduce (Chen et al. (2018)
46M
2.10M
7.79x
37.63
Structured Embedding (Shi (0188)
46M
2.07M
7.90x
37.78
Tensor Train Khrulkov et al. 201
46M
2.12M
7.72x
37.27
Distilled Embedding (Ours)
46M
2.08M
7.87x
37.78
Table I: Machine translation accuracy in terms of BLEU for WMT En-Fr on newstest2014.
approximately 6000 source tokens and 6000 target tokens for each GPU worker. All experiments were run using the TensorFlow framework
RESULTS
5.1 MACHINE TRANSLATION
We present BLEU score for method and compare it with SVD, GroupReduce (Chen et al. (2018), Structured Emedding (Shi Yu (2018)) Tensor Train (Khrulkov et al. (2019) and smaller transformer network with the all the methods except Tensor Train since was pointed out in Khrulkov et al.) (2019) that there no difference in performance between random initialization and tensor train learnt initializatinn. Once initialized we plug the decomposed embedding and fine-tune till convergence. None of the weights are frozen during fine-tuning.
number of parameters. We learn decomposition for
presents the results on English- translation. The hyper-parameters for tuning the com- peting methods are presented Section [..] of the Appendix. We see that on this task our method along with Structured Embedding performs the best. Group Reduce is the next, and SVD performs better than Tensor Train, showing that SVD strong baseline, when fine-tuned till convergence. We also compare against a layer neural network (NN) with the same parameterization as distilled embedding which has not been initialized offline. The results show that initializing the NN with the embedding weights is important.
English-Gerran translation, as seen in Table P our method outperforms all other methods. The smaller transformer network does well and is only surpassed by GroupReduce amongst the competing methods. SVD again performs better than Tensor Train.
We present the Portuguese-English translation results on Table 3 This task presents problem where the embedding matrix constitutes the majority of the parameters of the neural network. The embedding dimension is smaller (256) compared to the other two tasks but embedding compres- sion yields a BLEU score increase in all methods except Structured Embedding. This is due to a regularization effect from the compression. Our model again achieves the highest BLEU score.
On these three experiments we demonstrate that our funneling decomposition method with embed- ding distillation consistently yields higher BLEU scores compared to the competing methods.
5.2 ABLATION STUDY
We present different experiments to demonstrate the effect I) Model Initialization. 2) Embedding Distillation, 3) Fine-tuning strategies, 4) Compression capability and 5) Extension and generality of our method.
httss://www.tensofffoo....m

---

Under review as a conference paper at ICLR 2020
Emb.
Emb.
WMT En-De
Model
Param
Params
Compression Rate
(BLEU)
Transformer Base
63M
18.94M
1.0x
27.08
Smaller Transformer Network (400)
46M
14.8M
1.28x
26.72
SVD with rank 64
46M
2.40M
7.89x
26.32
End-to-End NN with non-linearity
46M
2.40M
7.89
26.14
GroupReduce (Chen et al. (2018)
46M
2.40M
7.88x
26.75
Structured Embedding (Shi (018))
46M
2.40M
7.89x
26.34
Tensor Train KKrulkov et al. 201
46M
2.44M
7.75x
26.19
Distilled Embedding (Ours)
46M
2.40M
7.89x
26.97
Table 2: Machine translation accuracy in terms of BLEU for WMT En-De on newstest2014.
Emb. Params
Emb. Compression Rate
IWSLT Pt-En (BLEU)
Model
Param
Transformer Small
11M
8.19M
1.0x
41.43
Smaller Transformer Network (136)
5M
4.35M
1.88x
40.71
SVD with rank 64
5M
2.06M
3.96x
42.37
End-to-End NN with non-linearity
5M
2.06M
3.96
42.27
GroupReduce (Chen et al. (2018)
5M
2.06M
3.96x
42.13
Structured Embedding (Shi (2018)
5M
2.06M
3.97x
41.27
Tensor Train Khrulkov et al. (2019)
5M
2.06M
3.96x
42.34
Distilled Embedding (Ours)
5M
2.06M
3.96x
42.62
Table 3: Machine translation accuracy in terms BLEU for IWSLT Pt-En.
Initialization We do an ablation study on all the three language pairs defined in Section 4.1 to conclude, if random initialization is better than model based initialization. We conclude that model based initialization, consistently performs better (Table 4
Embedding Distillation Table presents different compression rates on the Pt-En task, and em- bedding distillation performs better across all of them. In Table 4 we see that across all language pairs when we initialize our model using weights from the funneling decomposition, we improve when using Embedding Distillation during finetuning. We performed embedding distillation with random initialization only on the smaller Pt-En dataset and observed that Embedding Distillation improves BLEU score.
Compression Rate 15.86x with only a drop BLEU score for Pt-En.
We demonstrate in Table that it is possible to compress the embedding up to
Re-training Fine-tuning important component in our method and we demonstrate through our experiments that at convergence most of the techniques are close in performance. Table shows that freezing embedding weights and re-training the network weights or vice versa leads to sharp
Emb. Params
Emb. CR
Random Initialization
Model Initialization
Model Params
No Distillation
Emb. Distillation
No Distillation
Emb.
Distillation
En-Fr Fr
46M
2M
7.87x
37.04
37.54
37.78
En-De
46M
2M
7.89x
26.07
26.7
26.97
Pt-En
5M
2M
3.96x
42.29
42.36
42.5
42.62
Table 4: Comparison of different methods for Funneling (64), CR refers to the compression rate.

---

Under review as a conference paper at ICLR 2020
Emb. Params
Emb.
No
Emb. Distillation
Params
CR
Distillation
11M
8M
1.0x
41.43
5M
2M
3.96x
42.50
42.62
4M
IM
7.93x
42.44
42.60
4M
516k
15,86x
40.42
40.60
Table 5: Comparison of different compression rates with bottleneck sizes of 64, 32 and 16 accord- ingly for IWSLT Pt-En.
Model
BLEU
Model
BLEU
Proposal
42.60
GroupFunneling
42.52
embedding distillation
42.44
(Rand. Initialized + Emb. Distil.)
non-linearity
42.34
GroupFunneling
42.49
(Rand. Initialized)
Proposal (Freeze non-emb. weights)
33.34
GroupReduce
42.13
Proposal (Freeze emb. weights)
20.49
Table 7: GroupFunneling (i.e. GroupReduce Funneling) IWSLT Pt-En.
Table 6: BLEU score for IWSLT Pt-En with compression rate 7.93x.
drop in BLEU score, thus, need re- re-train all the weights. The use of non-linearity and adding embedding distillation also improves BLEU score after finetuning.
Extension We experimented with applying two key lessons from method, namely, using non-linear function and embedding distillation, to model initialized with group partitions of the GroupReduce method (Chen al., 2018), refer this method GroupFunneling Table shows that, GroupFunneling Funneling achieves higher BLEU score on Pt-En compared to GroupReduce.
DISCUSSION
Importance of Non-linearity We postulate that only a subset of word vector dimensions, explains most of the variance, for most word vectors in the embedding matrix. Thus, using ReLU activation might help in regularizing the less important dimensions for a given word vector.
Importance Reconstruction Loss We propose that the embedding reconstruction might suffer from adding the ReLU activation function. Thus, adding loss for embedding reconstruction helps in grounding the embedding and loose lot of information. Thus, the amount of regularization is controlled by the A.2 as reconstruction loss performs worse without the ReLU activation function.
Our intuition is partly justified by results shown in Table
CONCLUSION AND FUTURE WORK
In this paper we proposed Distilled Embedding, a low-rank matrix decomposition with non-linearity in the bottleneck layer for shared word-embedding and vocabulary projection matrix. We also in- troduce knowledge distillation the embedding during fine-tuning using the full embedding matrix as the teacher and the decomposed embedding the student. We compared our proposed approach with state-of-the-an art methods for compressing word-embedding matrix. We did extensive experi- ments using three different sizes of datasets and showed that our approach outperforms the ofa the art methods on the challenging task of machine translation. For future work, we will apply our approach compress beed-forward and multi-head attention layers of the transformer network.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Acharya, Rahul Goel, Angeliki Metallinou, and S. Inderjit Dhillon. Online embedding com- pression for text classification using low rank matrix factorization. AAAI, 2019.
Patrick Chen, Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. Groupreduce: Block-wise low- rank approximation for neural language model shrinking. In Advances in Neural Information Processing Systems, pp. 10988, 2018.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decompo- sition. SIAM journal on Matrix Analysis and Applications, 215:::22331222 2000.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1XX3::5503.02531 2015.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence. 33(1):117-128; 2010.
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019.
Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan Oseledets. Tensorized embed- ding layers for efficient model compression. arXiv preprint arXiv: 1000: 10787, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
412. 6980
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. CoRR, abs/1808.06226, 2018. URL
06225
Zhongliang Li, Raymond Kulhanek, Shaojun Wang, Yunxin Zhao, and Shuang Wu. Slim embedding layers for recurrent neural language models. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Tomáš Mikolov, Martin Karafiát, Lukáö Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
Andriy Mnih and Ruslan R Salakhutdinov. Probabilistic matrix factorization. In Advances in neural information processing systems, pp. 1257-1264, 2008
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Advances in neural information processing systems, pp. 442-450, 2015.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295- 2317, 2011.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 31 311-318. Association for Computational Linguistics, 2002.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regular- izing neural networks by penalizing confident output distributions. CoRR, abs/ abs//700 770.06548, 2017. 06548
URL http::/arxii org http://bs/110111

---

Under review as a conference paper at ICLR 2020
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL https:/ /www www.wawwww....gg/...... http://www..../aaaooooooooooooo.
-18-6319
Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608. 05859, 2016.
Adriana Romero, Nicolas Ballas, Samira E. Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference acoustics, speech and signal processing, pp. 6655- 6659. IEEE, 2013.
Jürgen Schmidhuber, Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
Kaiyu Shi and Kai Yu. Structured word embedding for low memory neural network language model. In Interspeech, pp. 1254-1258, 2018.
Raphael Shu and Hideki Nakayama. Compressing word embeddings via deep compositional code learning. arXiv preprint 171 2017.
Nathan Srebro and Tommi Jaakkola. Weighted low-rank approximations. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp- 720-727. 2003.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks. CoRR, 0bos/1499.225, 2014. UBL/ttp:://arrrvv.o////a//////
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shiens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CoRR, abs/1512.00567, 2015. URL
http::/aree org/ 1at.bb///1111005061
Xu Tan, Yi Ren, Di He, Tao Qin. Zhou Zhao, and Tie- Yan Liu. Multilingual neural machine trans- lation with knowledge distillation. CoRR, 1ess/1922.1461,, 2019. [tllatt://axxi..ot
abs/1902. 10461
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL pdf
http:://pppes httpppppprr...aascppppppe/ee.e httppp/cccpppppeeeeeeeeeeeeraoaa
Genta Indra Winata, Andrea Madotto, Jamin Shin, and Elham Barezi. Low-rank matrix factoriza- tion of Istm effective model compression. 2018
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 ADDITIONAL. HYPER-PARANEE PARAMETERS
WMT En-Fr Smaller Transformer Network denotes a network with the same configuration as Transformer Base but with hidden size dmodel of 416. For GroupReduce, to match the same com- pression rate used number of clusters being equal to 10 and minimum rank mimn be 22. For SVD, we decided to set the rank to 64. For Tensor Train, we set the embedding shape to be [25, 32, 40] x x[8, 8] and the Tensor Train Rank be 90. For structured embedding we use group size as 32 and number of clusters as 2048, we then use the quantization matrix and learn the clusters from scratch.
WMT En-De Smaller Transformer Network denotes a network with the same configuration as Transformer Base but with hidden size dmmdee of 400. For GroupReduce, to match the com- pression rate we used number of clusters с being equal to 10 and minimum rank Tmin to be 23. For SVD, we decided
the rank 64. For Tensor Train, set the embedding shape to be [25, 37, 40) x [8, 8, 8] and the Tensor Train Rank to be 90. For structured embedding we use group size 32 and number of clusters as 2376, we then use the quantization matrix and learn the clusters from scratch.
IWSLT Pt-En Smaller Transformer Network denotes a network with the same configuration as Transformer Small but with hidden size dmodel of 136. For GroupReduce, to match the same com- pression rate we used number of clusters c being equal to 15 and minimum rank mmin be 30. For SVD, we decided to set the rank to 64. For Tensor Train, we set the embedding shape to be [25, 32, 40) x (8, 4, 8] and the Tensor Train Rank to be 125. For structured embedding we use group size 32 and number of clusters as 4048, we then use the quantization matrix and learn the clusters from scratch.
A.2 PARAMETER COUNT
Table A. presents the the number of parameters in the different transfomer layers for the transformer
base architecture.
Parameters
Embedding
FFN
Multi- head attention
Linear
Number
26M
25M
14M
5M
Percentage
37%
36%
20%
7%
Table A.I: Parameters in the Transformer Base model Vaswani et 2017 based 50k dic- tionary size and tied input and output embedding.
A.3 SVD WITH RECONSTRUCTION LOSS
Table A.2 presents BLEU scores for SVD on English French translation when combining recon- struction loss with SVD. This result suggests that SVD does not perform well with reconstruction loss.
Model
BLEU
SVD with rank 64
37.44
SVD with rank 64 with Recon. Loss (alpha 0.1)
37.29
Table A.2: Comparison of SVD with and without reconstruction loss on En-Fr translation.
A.4 CO-RELATION OF RECONSTRUCTION Loss AND TRANSLATION
We present the L2 reconstruction losses for factorization of embedding matrices table A.3 We that lower reconstruction loss does not necessarily lead to better translation performance. e.g.
11

---

Under review as a conference paper at ICLR 2020
GroupReduce has the lowest reconstruction loss on Pt-En but Funneling based solution gets higher BLEU score.
En-Fr Recon, Loss
En-De Recon. Loss
Pt-En Recon. Loss
Model
SVD
2.247
2.447
1.199
GroupReduce
2.247
2.445
1.173
Funneling
2.238
2,441
1.194
Table A.3: Reconstruction losses for all language pairs.
A.5 HYPER-PARAMETER SENSITIVITY
We present the sensitivity of our algorithm to the hyper-parameter for Pt-En translation in ta- ble A.4 We observe that the algorithm works well over wide range of a values. In this work we did not tune a for different datasets but doing might yield higher performance as is the case for Pt-En translation with a = 0.5.
BLEU
42.5
0.01
42.62
0.1
42.65
0.3
42.66
0.5
42.72
0.7
42.57
0.9
42.03
Table A.4: BLEU score for Pt-En translation for different alpha values.
12


Reviews:
[{"review_id": "HJl-Mn5kcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use low-rank matrix decomposition for embedding compression, with relu in the reconstruction layer to gain non-linearity. Experiments on machine translation task shows improvement compared with state-of-the-art methods with different compression rates.\n\nDetailed comments:\n1)\tThe technical contribution seems to be a bit limited. Using relu in the reconstruction function looks straightforward and adding reconstruction loss in objective function is also common practice. Also, not much insight is provided on why such approach works better than other baselines. \n\n2)\tExperiments:\na.\tIt is good to see such simple approach outperforms several more sophisticated baseline methods. Also, ablation study is also performed to show the effect of different components.\n\nb.\tHow does the time complexity and running time of the proposed method compared to the baselines?\n\nc.\tThe paper only evaluates distilled embedding on one task (i.e., machine translation). The experiments would be more convincing if evaluated on more tasks as well.\n\nd.\tIt could be helpful to include some sensitivity analysis on the hyperparameters such as \\alpha which controls the weight of reconstruction loss. \n\nIn conclusion, this paper seems to be below the bar and I would recommend a ‘weak reject’ for the paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgi5BU8qS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There are many ways to reduce the memory footprint and increase speed of a neural network: weight quantisation, compression, coarse-to-fine, knowledge distillation, etc. The method proposed in this work is a specific case of knowledge distillation that focuses on the discrete-input-to-first-layer and output-layer-to-discrete-output transformations, which represent a large portion of the parameters.\n\nThe authors propose to use a variant of SVD (which can be viewed as 2 linear transformation, with a middle dimension that represents an embedding), where the first transformation is linear with a ReLu, and the second is linear. By approximating the learned matrices of the model, the experiments show that using the proposed variant of SVD gives similar predictive performance compared to the original model, with a fraction of the parameters.\n\nHowever, it seems that the authors could have simply replaced the input by a 2-layer NN (first a linear+ReLu, then a Linear) to obtain the same parametrisation, but they could have learned the parameters in a end-to-end fashion. It is not clear to me why using a surrogate L2 loss within the model should give better predictive performance than a fully end-to-end trained neural network. Without this comparison, I do not think the proposed experiments are conclusive enough.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lOWHeJ9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nThis paper proposes a method for compressing embedding matrices of both encoder/decoder embeddings.\nThe basic idea of the proposed method is to reconstruct the embedding matrix by what they called the “funneling decomposition” method, whose parameter shape is identical to the SVD (low-rank matrix) decomposition with additional non-linear function.\nTherefore, the idea itself is not so novel and innovative.\nMoreover, their method requires the embedding matrix as the teacher signal for calculating the reconstruction loss.\nWe need to note that the memory requirement of the proposed method during training will increase.\n \nOne of the notable advantages of the proposed method is that their proposed method seems to successfully reduce the embedding matrix even if it shares the parameters with the output layer, which is a de-facto standard model architecture for NMT.\nAs pointed out by the authors, this seems to be the first success of reducing the embedding matrix with a tied embedding setting.\n\n\n1,\nThe authors claim that “We demonstrate that at the same compression rate our method outperforms existing state-of-the-art methods.” at the end of the Introduction section.\nHowever, according to Tables 1, 2, and 3, it seems that the performance gain is marginal compared with similar methods.\nFor example, \n37.78 (proposed) <=> 37.78 (Shi & Yu (2018)) \n26.97 (proposed) <=> 26.75 (Chen et al. (2018)\nand\n 42.62 (proposed) <=> 42.37 (SVD with rank 64),\nwhich are the at most 0.25 BLEU gain.\nI believe that most of MT researchers hardly say that BLEU 0.25 difference is a significant improvement. Besides, the authors should perform a statistically significant test if they say “our method outperforms existing state-of-the-art methods.”\n \n \n2\nI am a bit confused about the following inconsistency;\nThe authors say that “Compressing embedding matrices without sacrificing model performance is essential for successful commercial edge deployment” in the abstract.\nHowever, according to Table 1, the number of parameters for embeddings is 16.3M, which is only 27% of the total number of parameters in Transformer base.\nBy this fact, compressing embedding matrices seems not essential for successful commercial edge deployment.\n \nIn Table 6, it is explicitly unclear what is the difference between \n“Funneling with Emb. Distillation”, “Funneling (with non-linearity),” and “Funneling (with retraining all weights).”\nPlease give us a more precise explanation.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to further distill token embeddings via what is effectively a simple autoencoder with a ReLU activation. All reviewers expressed concerns with the degree of technical contribution of this paper. As Reviewer 3 identifies, there are simple variants (e.g. end-to-end training with the factorized model) and there is no clear intuition for why the proposed method should outperform its variants as well as the other baselines (as noted by Reviewer 1). Reviewer 2 further expresses concerns about the merits of the propose approach over existing approaches, given the apparently small effect size of the improvement (let alone the possibility that the improvement may not in fact be statistically significant).

Decision: Reject