Tensorized Embedding Layers for Efficient Model Compression
Authors: Oleksii Hrinchuk
Valentin Khrulkov
Leyla Mirvakhabova
Ivan Oseledets
Venue:  (2020)
Keywords: Embedding layers compression
tensor networks
low-rank factorization

Abstract: The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance.  We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

Under review conference paper at ICLR 2020
TENSORIZED EMBEDDING LAYERS FOR EFFICIENT MODEL COMPRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for wide range of architectures, from MLPs to LSTMs and Transformers.
INTRODUCTION
Deep neural networks (DNNs) typically used in natural language processing (NLP) employ large embeddings layers, which map the input words into continuous representations and usually have the form of lookup tables. Despite such simplicity and, arguably because of it, the resulting models are cumbersome, which may cause problems in training and deploying them in limited resource setting. Thus, the compression of large neural networks and the development of novel lightweight architectures have become essential problems in NLP research.
One way to reduce the number of parameters in the trained model is to imply a specific structure on its weight matrices (e.g., assume that they are low-rank can be well approximated by low-rank tensor networks). Such approaches are successful at compressing the pre-trained models, but they do not facilitate the training itself. Furthermore, they usually require additional fine-tuning stage to recover the performance of the original model.
In this paper, introduce new, parameter efficient embedding layer, termed TT-embedding, which can be plugged in into any model and trained end-to-end. The benefits of our compressed TT-layer twofold. Firstly, instead of storing huge embedding matrix, store sequence of much smaller 2-dimensional and 3-dimensional tensors, necessary for reconstructing the required embeddings, which allows compressing the model significantly at the cost of negligible performance drop. Secondly, the overall number of parameters can be relatively small (and constant) during the whole training stage, which allows use larger batches or train efficiently
case of limited resources.
To validate the efficiency of the proposed approach, we have tested it on several popular NLP tasks. In our experiments, we have observed that the standard embeddings can be replaced by TT-embeddings with the compression ratio of 3 orders without any significant drop (and sometimes even with slight gain) of the metric of interest. Specifically, we report the following compression ratios of the embedding layers: 441 on the IMDB dataset with 0.2% absolute increase in classification accuracy; 15 on the WMT 2014 En-De dataset with 0.3 drop in the BLEU score.
Additionally, we have also evaluated our algorithm on task of binary classification based large number of categorical features. More concretely, we applied TT-embedding to the click through rate (CTR) prediction problem, crucial task in the field of digital advertising. Neural networks, typically used for solving this problem, while being rather elementary, include a large number of embedding layers of significant size. As result, majority of model parameters that represent these layers, may occupy hundreds of gigabytes of space. We show that TT-embedding not only considerably reduces the number of parameters in such models, but also sometimes improves their accuracy.

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
In recent years, large body of research was devoted to compressing and speeding up various components of neural networks used in NLP tasks. Joulin et al. (2016) adapted the framework of product quantization to reduce the number of parameters in linear models used for text classification. See et al. (2016) proposed to compress LSTM-basse neural machine translation models with pruning algorithms. Lobacheva et al. (2017) showed that the recurrent models could be significantly sparsified with the help of variational dropout (Kingma et al., 2015). Chen et al. (20) (2018b) proposed more compact K-way D-dimensional discrete encoding scheme replace the "one-hot" encoding of categorical features, such as words in NLP taks. Very recently, Chen et al. (2018a) and Variani et al. (2018) introduced GroupReduce and WEST, two very efficient compression methods for the embedding and softmax layers, based on structured low-rank matrix approximation. Concurrently. Lam (20 8) proposed the quantization algorithm for compressing word vectors and showed the superiority of the obtained embeddings on word similarity, word analogy, and question answering tasks.
Tensor methods have also been already successfully applied to neural networks compression. Novikov et al. (2015) coined the idea of reshaping weights of fully-connected layers into high-dimensional tensors and representing them in Tensor Train (TT) (Oseledets, 2011) format. This approach was later extended to convolutional (Garipov et al., 2016) and recurrent (Yang et al., 2017; Tjandra et al., 2017; Yu et al., 2017) neural networks. Furthermore, Lebedev et al. (2015) showed that convolutional layers could be also compressed with canonical (CP) tensor decomposition (Carroll & Chang, 1970; Harshman, 1970). Finally, Wang et al. (2018) compressed both fully-connected and convolutional layers with Tensor Ring decomposition (Zhao et al., 2016). While all these methods allowed to reduce the number of parameters in the networks dramatically, they mostly capitalized on heavy fully- connected and convolutional layers (present in AlexNet (Krizhevsky et al., 2012) or VGG (Simonyan & Zisserman, 2014)), which became outdated in the following years, Recently, Ma et al. (2019) succesfully applied Block-Term Tensor Decomposition to the compression of self-attention modules in the Transformer (Vaswani et al., 2017) architecture. In this work, we show the benefits of applying tensor machinery to the compression of embedding layers, which are still widely used in NLP.
TENSOR TRAIN EMBEDDING
In this section, we briefly introduce the necessary notation and present the algorithm for training the TT "T-embedding layer. Hereinafter, by N N-way tensor X' we mean a multidimensional array:
X E g/ihxxxxxxxxxx
with entries X ( in), such that (0s is < I
3.1 MOTIVATION
Since most of the parameters in the NLP models occupy the embedding layers, greatly reduce size of the entire model by compressing these layers. Our goal to replace the standard embedding matrix with more compact, yet powerful and trainable, representation which would allow to efficiently map words into vectors.
The simplest approach to compactly represent matrix of large size is to the low-rank matrix factorization, which treats matrix E E / as a product of two matrices E = UVT. Here U E / and V IR / R are much "thinner" matrices, and R is the rank hyperparameter. Note that rather than training the model with the standard embedding layer, and then trying to compress the obtained embedding, we can initially seek the embedding matrix in the described low-rank format. Then, for evaluation and training, the individual word embedding Eli. :] can be computed as product Uli which does not require materializing the full matrix E. This approach reduces the number of degrees of freedom in the embedding layer from IJ to (I + J.
However, typically, in the NLP tasks the embedding dimension is much smaller than the vocabulary size 7, and obtaining significant compression ratio using low-rank matrix factorization is problematic. In order to preserve the model performance, the rank R cannot be taken very small, and the compres- IJ IJ
sion ratio is bounded by 7I+JJR < k. which is close to for usually full-rank embedding matrix (see Figure in Chen al. (2018b)). To overcome this bound and achieve significant compression
2

---

Under review conference paper at ICLR 2020
ratio even for matrices of disproportional dimensionalities, we reshape them into multidimensional tensors and apply the Tensor Train decomposition, which allows for more compact representation, where the number of parameters falls down to logarithmic with respect 7.
3.2 TENSOR TRAIN DECOMPOSITION
A tensor X is said to be represented in the Tensor Train (TT) format (Oseledets, 2011) if each element of can be computed as:
R1
R₂
Ry- Σ TN-1=1
Eii .....,...................... 1a)
g (
where the tensors (8)) - are the so-called TT-cores and Rg Ry by definition. The minimal values of [Rx] ] N-1 for which the IT-decomposition exists are called TT-ranks. Note, that the element X ( i2 2N) is just effectively the product of 2 vectors and N 2 matrices:
X Xii,,... iN) = (2) [in, g 1, 22.
( g (N) [E. inl.
1x R2
R2
Ry- Ry-3XRRy 2
Ry-1x1
where g [1, i stands for the slice (a subset of tensor with some indices fixed) of the corre- sponding TT-core (i)).
The number of degrees of freedom in such decomposition be evaluated as Σ k=1 RE-1 T Rg- Thus, in the case of small ranks, the total number parameters required to store tensor in TT- representation is significantly smaller than I 4 parameters required to store the full tensor of the corresponding size. This observation makes the application of the TT TT-decomposition appealing in many problems dealing with extremely large tensors.
3.3 MATRIXX
Let X RIXJ be a matrix size natural numbers, = -i=1
Given two arbitrary factorizations its dimensions into -I[ TR and J = =II Ja. we can reshape' and transpose this matrix into an N-way tensor X' E xp1xxxxlaxx....xxxxxx>. and then apply the IT-decomposition to it, resulting in more compact representation.
More concretely, define the bijections I(i) = (i1,...,4y iN) and J()) = ,. jNn that map row and column indices and of the matrix X to the N-dimensional vector-indices such that 0 < is < Ix. 0 S jk < Jk. Vk = 1,..., .. From the matrix X we can form an N N-way tensor X' whose k-th dimension of length JgJ and indexed by the tuple (kx. This tensor is then represented in the TT format:
(1) Such representation of the matrix in the TT-format is called TT-matrix (Oseledets, 2010: Novikov et al., 2015) and also known as Matrix Product Operator (Pirvu et al., 2010) in physics literature. The factorizations (T, ... IN) x J2.... JN) will be referred as the shape of TTT-marrxx, or 77-shapes. The process of constructing the TT-matrix from the standard matrix is visualized in Figure for the tensor of order 3. Note, that this the TT-cores in fact 4-th order tensors, but all the operations defined for tensors in the TT-format are naturally extended to TT-matrices.
X ((1,)1)) (iN.JN)) = g (2) (11)))): ....:.............. ....:)))......))))) (
(iN-iN)]-
3.4 TT-EMBEDDING
By emembddding we call layer with trainable parameters (TT-cores) represented TT-matrix E of the underlying tensor shape (4, I2....Iy
which can be transformed into
valid embedding layer E IxJ with 1 I TT- one has also to provide the TT-ranks, which are treated as hyperparameters of the layer and explicitly define the total compression ratio.
k== Ja and J IT Jr. To specify the shapes of
by reshape mean column-major reshape command such numpy. reshape in Python.

---

Under review as a conference paper at ICLR 2020
Reshaping
TT decommposition
4 1 x
1 xJ
Ix4
R1
R2
1x
1s x J3
R;
12 J22
J1 X
1 x J2
Figure 1: Construction of the how the single element in the initial matrix is transformed into the product of the highlighted vectors and matrices in the To-cores.
from the standard embedding matrix. Blue color depicts
In order to compute the embedding for a particular word indexed i in the vocabulary, we first map the row index i into the N N-dimensional vector index (i1...., ix) ), and then calculate components of the embedding with formula (1). Note, that the computation of all its components is equivalent
to selecting the particular slices in TT-cores (slices of shapes J1 R in g(), Rí x J2 x R₂ in ((2) and so on) and performing sequence of matrix multiplications, which is executed efficiently in modern linear algebra packages, such as BLAS. Pseudocode for the procedure of computing the mapping + (f1,-..;3 ; is given in Appendix A.
In order to construct TT embedding layer for a vocabulary of size I and embedding dimension J, and to train model with such a layer, one has to perform the following steps.
Provide factorizations of T and J into factors T 11 x 12 x IN and x ... x JN. and specify the set of TT-ranks [R, R2,..., RN-1}.
J = J1 x J1
Initialize the set of parameters of the embedding θ = {g(h) E IR x NN 1k=1
(R)
Concrete initialization scenarios are discussed further in the text.
During training, given a batch of indices f2, it). compute the corresponding embed- dings (e1, c 0%) using Eq. (1) and Algorithm 1.
Computed embeddings can be followed by any standard layer such as LSTM (Hochreiter Schmidhuber, 1997) self-attention (Vaswani et al., 2017), and trained with backpropaga- tion since they differentially depend on the parameters O.
TT-embedding implies a specific structure on the order of tokens in the vocabulary (the order of rows in the embedding matrix), and determining the optimal order is an appealing problem solve. However, we leave this problem for future work and use the order produced by the standard tokenizer (sorted by frequency) in our current experiments.
We also experimented with a more general form of TT-decomposition, namely Tensor Ring decompo- sition (Zhao et al., 2016; Wang al., 2018). This decomposition by construction has the appealing property of being circular permutation invariant (and, thus, more robust with respect to the order of the tokens), which could have potentially provided an improvement over the TT-based models with simple frequency based ordering. Our experiments with TR decomposition on Transformer for NMT can be found in Appendix B.
Initialization The standard way to initialize an embedding matrix E E RRXx is via, e.g., Glorot initializer (Glorot& Bengio, 2010), which initializes each element as E(i,j) ~ N (0. ( For the TT-embedding, can only initialize the TT-cores, and the distribution of the elements of the resulting matrix & is rather non trivial.. However, it is easy to verify that we initialize each (0, 1), the resulting distribution of the matrix elements -==
TT-core element as (A)) (rk-1, 42, ~ (0,
{i,j) has the property that E(E(i, observation, in order to obtain the desired variance Var (i. = =o² while keeping E(E (i.))) = 0.
0 and Var(E(i.j)) - Rs- R². Capitalizing this
we can simply initialize each TT- as
9/N
g (rR ~
(2)
4

---

Under review conference paper at ICLR 2020
The resulting distribution is not Gaussian, however, it approaches the Gaussian distribution with the increase of the TT-rank (Figure 2).
TT-rank
8
16
-3 -2
2
3
Distribution domain
Figure 2: Distribution of matrix elements of the TT- of shape (5, 5, (5, 5,5,) initialized by formula (2) with = 1. As the TT-rank increases, the resulting distribution approaches ((0,, 1).
In our experiments, we have used the modified Glorot initializer implemented by formula (2). which greatly improved performance, as opposed to initializing TT-cores simply via standard normal distribution. It is also possible to initialize TT-embedding layer by converting the learned embedding matrix into TT-format using the TT-SVD algorithm (Oseledets, 2011), however, this approach requires the pretrained embedding matrix and does not exhibit better performance in practice.
Hyperparameter selection Our embedding layer introduces two additional structure-specific hy- perparameters, namely TT-shapes and TT-ranks.
TT-embedding does require the vocabulary size / to be represented exactly the product of factors J,.... JN, in fact, any factorization IT I 7 N will suffice. However, in order to achieve the highest possible compression ratio for fixed value of I, the factors { } should be as close to each other possible. Our implementation includes a simple automated procedure for selecting good values of { during TT em-embddinng initialization. The factors J....., JN are which can easily chosen to support good factorization, e.g., 512 8 8 x 8 or 480 6 5 x 4 4.
k=
defined by the embedding dimensionality
The values of TT-ranks directly define the compression ratio, so choosing them to too small or too large will result into either significant performance drop or little reduction of the number of parameters. In our experiments, we set all TT-ranks to 16 for the problems with small vocabularies and 64 192 for the problems with larger vocabularies, which allowed us achieve significant compression of the embedding layer, at the cost of a tiny sacrifice in the metrics of interest.
EXPERIMENTS
Code We have implemented TT embeedidss described in Section in Python using PyTorch (Paszke et al., 2017). The code is available at the anonymous repository https///githb..om//t embeeddng/tt-mmeeddigg..
Experimental setup
We tested our approach on several popular NLP tasks:
Sentiment analysis as starting point in our experiments, we test TT-embeddings rather simple task of predicting polarity of a sentence.
Neural Machine Translation (NMT) - to verify the applicability of TT-embeddings in more practical problems, we test on a more challenging task of machine translation.
Language Modeling (LM) then, we evaluate TT-embeddings on language modeling tasks in the case of extremely large vocabularies.
5

---

resulting in Under review as a conference paper at ICLR 2020
Click Through Rate (CTR) prediction - finally, we show that TT-embeddings can be applied for the binary classification with categorical features of significant cardinality.
To prove the generality and wide upplicability of the proposed approach, we tested it on various architectures, such as MLPs (CTR), LSTMs (sentiment analysis), and Transformers (NMT, LM).
Note that Transformers in LM and NMT use the same weight matrix for their embedding and softmax layers (Press & Wolf, 2016; 6 Inan et al., 2016) which already significantly reduces model size. Untying weights and tensorizing the embedding layer only will lead to the increase in the number of parameters instead of compression. In our experiments, we use two separate ITTdecomposiiiins of the same shape for embedding and softmax layers and report the compression ratios
[V| Vlddd pxramm
4.1 SENTIMENT ANALYSIS
For this experiment, have used the IMDB dataset (Maas et al., 2011) with two categories, and the Stanford Sentiment Treebank (SST) (Socher et al., 2013) with five categories. We have taken the most frequent 25000 words for the IMDB dataset and 17200 for SST, embedded them into J J-dimensional space using either standard embedding or TT-embedding layer, and performed classification using a standard bidirectional two-layer LSTM with hidden size h 128, and dropout rate Parop 0.5.
Our findings are summarized in Table 1. We observe that the models with largely compressed embedding layers can perform equally even better than the full uncompressed models. This suggests that learning individual independent embeddings for each particular word is superftuous, as the expressive power of LSTM is sufficient to make use of these intertwined, yet more compact embeddings. Morcover, slightly better test accuracy of the compressed models in certain cases (e.g., for the SST dataset of rather small size) insinuates that imposing specific tensorial low-rank structure on the embedding matrix can be viewed as a special form of regularization, thus potentially improving model generalization. A detailed and comprehensive test of this hypothesis goes beyond the scope of this paper, and we leave it for future work.
Table I: Sentiment analysis, LSTM on IMDB and SST datasets. Embedding compression is calculated as the ratio between the number of parameters in the full embedding layer and TT TT-embedding layer. The LSTM parts are identical both models, and the TT-ranks were set to 16 in these experiments.
Emb
Total
Dataset
Model
Embedding shape
Test acc.
compr.
params
Full
25000 x 256 (25, 30, 40) (4, 84) (10 15, 20) x (4, 44,4,4,4)
0.886
1
7.19M
TTI
0.871
93
0.86M
IMDB
TT2
0.888
232
0.82M
TT3
(5, 5, 6,6,) 2,,2,2,2,444)
0.897
441
0.81M
Full
17200 x 256
0.374
1
5.19M
TTI
(24, 25, 30) (4,8,8) (10, 10, 12, 15) 10,
0.415
78
0.85M
SST
TT2
x
0.411
182
0.82M
TT3
54,5,5,5,6,6) 6, x (2,2,,,,)
0.399
307
0.81M
4.2 NEURAL MACHINE TRANSLATION
For this experiment, we have trained the Transformer-big model (dmodal 1024, du = 4096, h = 16) from (Vaswani et al., 2017) on WMT 2014 English-German dataset consisting of roughly 4.5 million sentence pairs. We evaluated on newstest201 4 dataset using beam search with beam size of 4 and no length penalty. We did not employ checkpoint averaging and used the last checkpoint to compute the BLEU score. Sentences were tokenized with joint vocabulary of 32768 tokens. For the full list of hyperparameters, see Appendix C.
tyte-pair-encodine
Our results summarized in Table 2. We observe that even in this rather challenging task, both embedding and softmax layers can be compressed significantly, at the cost of a small drop in the

---

Under review conference paper at ICLR 2020
BLEU score. However, with the increase of compression factor, the performance deteriorates rapidly, Compared to the sentiment analysis, NMT is a much more complex task which benefits more from additional capacity (in the form of more powerful RNN or more transformer blocks) rather than regularization (Bahdanau al., 2014; Vaswani et al., 2017; Wu et al., 2019), which may explain why we did not manage to improve the model by regularizing its embedding layers.
TT-embeddings induce 8% training iteration time overhead if compared to the baseline Transformer- big due to our current implementation heavy relying on slow torch einsum function while standard embedding and softmax layers make of fast and highly-optimized Tensor Cores for mixed peeediioeissio training. We expect a dedicated CUDA kernel to be much more efficient.
Table 2: NMT. Transformer-big on WMT English-to-Gemee dataset. Both case-sensitive
tokenized BLEU and de-tokenized SacreBLEU (Post, 2018) on newstest2014 are reported.
TT
Token BLEU
Sacre BLEU
Emb compr.
Total
Iter
Model
Embedding shape
rank
params
time
Big
32768 X 1024
29.58
28.84
210M
1.14
Big+TTI
32,32,32 32)
(8, 8, 16)
64
29.17
28.53
15.3
179M
1.23
Big+TT2
(32, 32)
X (8,8, 16)
48
28.53
27.97
26.8
178M
1.22
Big+TT3
(32, 32,32) 32)
(8,8, 16)
32
28.26
27.70
58.5
177M
1.22
4.3 LANGUAGE MODELING
We took the Transformer-Xi (Dai al., 2019). open source' state-of-the-an thr- language modeling architecture at the time this writing, and replaced its embedding and softmax layers with TT- factorizations. Then, tested different model configurations on the Wik (Merity et al., 2016) dataset and reported the results Table For the full list of hyperparameters, see Appendix C.
Compared to sentiment analysis and NMT, for embedding and softmax layers in LM. However, even moderate 3.8 times compression allowed us save 100M of weights at the cost of 1.5 perplexity drop.
not able to achieve that high compression ratios
Table 3: LM, Transformer- XL (Dai et al., 2019) on WikiText-103 dataset.
TT rank
Valid PPL
Test PPL
Emb compr.
Total params
Model
Embedding shape
TXL
267735 x 512 (60, 60, 75)
22.55
24.37
1
192M
TXL+TTI
(8,8,8)
192
24.38
25.67
3.8
94M
TXL+TT2
(60, 60, 75) x (8, 8,8
128
25.53
26.73
8.6
73M
TXL+TT3
(60, 60, 75)
8,,8,8)
96
26.73
28.04
15.1
65M
4.4 CLICK THROUGH RATE PREDICTION
Among other applications of the TT- "T-embedding layer. we chose to focus on CTR prediction, popular task in digital advertising (He et al., 2014). We consider open dataset provided by Criteo for Kaggle Display Advertising Challenge (Criteo Labs, 2014) which consists of 39 categorical features, 45.8M samples and is binary labeled according to whether the user clicked on the given advertisement. Unique values of categorical features are bijectively mapped into integers. To reduce the memory footprint, if the size this dataset is of order 10"), these integers are further hashed by taking modulus with respect to some fixed number such as 105. However, due to strong compression properties of TT-embbeddings, this is necessary for approach, and consider both full and hashed datasets in our experiments.
corresponding vocabulary is immense (e.g., cardinality some features in

---

Under review conference paper at ICLR 2020
CTR with the baseline algorithm The task hand can be treated as binary classification problem. As baseline algorithm, we consider the neural network with the following architecture. First, each of the categorical features passed through separate embedding layer with embedding size J. After that, the embedded features are concatenated and passed through 4 fully-connected layers of 1024 neurons and ReLU activation functions. In all experiments, we used Adam optimizer with the learning rate equal to 0.0005. Since many input features have large number of unique values (e.g., 10131227) and storing the corresponding embedding matrices would be costly, we employ the hashing procedure mentioned earlier.
CTR with TT-embeddings We substitute the embedding layers with the TT-embedding layers. Besides that, we leave the overall structure of the neural network unchanged with the same parameters as in the baseline approach. Table presents the experimental results on the Critco CTR dataset. To the best of our knowledge, our loss value is very close to the state-of-the-art result (Juan et al., 2016). These experiments indicate that the substitution of large embedding layers with TT TT-embeddings leads to significant compression ratios (up to 2011 times) with a slight improvement in the test loss, and up to 4200 with a small drop in the test loss. The total size of the compressed model does not exceed 20 Mb, while the baseline model weighs about 160 Mb. The obtained compression ratio suggests that the usage of TT-embedding layers may be beneficial in CTR prediction tasks.
Table 4: CTR prediction. The hashed dataset is constructed as specified Section 4.4 with hashing value 10a. Embedding layers with more than 2000 unique tokens were replaced by TT-embeddings with shape factorizations consisting of or 4 factors.
TT
Hidden size
Test loss
Emb. compr.
Total params
Hash
Model
Factorization
rank
Full
1024
0.4440
41.2M
TT1
factors
16
1024
0.4433
4.7M
10
TT2
factors
16
1024
0.4440
92
4.5M
TT3
factors
128
0.4515
2100
0.53M
TT4
4 factors
128
0.4530
4193
0.53M
TTI
factors
16
1024
0.4444
1004
5.2M
TT2
4 factors
16
1024
0.4438
2011
4.7M
DISCUSSION AND FUTURE WORK
We propose novel embedding layer, the TT-embedding, for compressing huge lookup tables used for encoding categorical features of significant cardinality, such as the index of a token in natural language processing tasks. The proposed approach, based on the TT-decomposition, experimentally proved to be effective, as it heavily decreases the number of training parameters at the cost of a small deterioration in performance. In addition, method be easily integrated into any deep learning framework and trained via backpropagation, while capitalizing on reduced memory requirements and increased training batch size.
Our experimental results suggest several appealing directions for future work. First of all, TT- embeddings impose concrete tensorial low-rank structure on the embedding matrix, which was shown to improve the generalization ability of the networks acting as a regularizer. The properties and conditions of applicability of this regularizer are subject to more rigorous analysis. Secondly, unlike standard embedding, we can introduce non-linearity into TT-cores to improve their expressive power (Khrulkov al., 2019). Additionally, it important to understand how the order of tokens in the vocabulary affects the properties of the networks with TT- TT-embedding, We hypothesize that there exists the optimal order tokens which better exploits the particular structure of TT-embedding and leads to boost in performance and/or compression ratio. Finally, the idea of applying higher-order tensor decompositions to reduce the number of parameters in neural nets is complementary to more traditional methods such pruning (Han al., 2015) and quantization (Hubara et al., 2017; Xu et al., 2018). Thus, would be interesting to make a thorough comparison of all these methods and stronger compression.
investigate whether their combination may lead
8

---

Under review conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: I009.04794 2014.
J Douglas Carroll and Jih-Jie Chang, Analysis of individual differences in multidimensiooal scaling via an n-way generalization of Eckart-Youn decomposition. Psychometrika, 35(3), 1970.
Patrick H Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-jui Hsieh. GroupReduce: Block-wise low-rank approximation for neural language model shrinking, NIPS, 2018a.
Ting Chen, Martin Renqiang Min, and Yizhou Sun. Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations, arXiv preprint arXiv:1806.09464, 2018b.
Criteo Labs. Kaggle Display Advertising Challenge, 2014. URL httph::tp///// httpps:/www....e. http:::://wwww..ggll...... com/
Zihang Dai, Zhilin Yang, Yiming Yang. William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.00 0008000000,...... 2019.
Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate tensorization: compressing convolutional and FC layers alike. arXiv preprint arXiv:1611.03214 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pp. 249-256, 2010.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135-1143, 2015.
Richard A Harshman. Foundations of the PARAFAC procedure: Models and conditions for an" explanatory" multimodal factor analysis. UCLA Working Papers in Phonetics, 1970.
Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. Practical lessons from predicting clicks on ads facebook. In Proceedings of the Eighth International Workshop Data Mining for Online Advertising, pp. 1-9.
ACM, 2014.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-ia,iv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18(187):1-30. 2017.
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint (tti://611.01666 2016.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models. arXiv preprint drXiv:1612.03651, 2016.
Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. Field-aware factorization machines for CTR prediction. In Proceedings of the 10th ACM Conference on Recommender Systems, pp. 43-50. ACM, 2016.
Valentin Khrulkov, Oleksii Hrinchuk, and Ivan Oseledets. Generalized tensor models for recurrent neural networks. arXiv preprint 1rXiv:1901.10801 2019.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems, pp. 3575-2583, 2015.

---

Under review conference paper at ICLR 2020
Alex Krizhevsky, 1lya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, 2012.
Maximilian Lam. Word2Bit-qqantized word vectors. arXiv preprint arXiv:1803.05665/ 2018.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding- up convolutional neural networks using fine-tuned CP-decomposition. ICLR, 2015.
Ekaterina Lobacheva, Nadezhda Chirkova, and Dmitry Vetrov. Bayesian sparsification recurrent neural networks. arXiv preprint arXiv: 00077, 2017.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Dawei Song, and Ming Zhou, tensorized transformer for language modeling. arXiv preprint arXiv: urXiv:1906.09777 2019.
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arviv: 1609.0760 2016.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov, Tensorizing neural networks. In Advances in Neural Information Processing Systems, pp. 442-450, 2015.
Ivan V Oseledets. Approximation of 2 x 2d matrices using tensor decomposition, SIAM Journal on Matrix Analysis and Applications, 31(4))210-21455 2010.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):
2295-2317, 2011.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NeurlPS 2017 Autodiff Workshop, 2017.
Bogdan Pirvu, Valentin Murg. J Ignacio Cirac, and Frank Verstraete. Matrix product operator representations. New Journal of Physics, 12(2):025012, 2010.
Matt Post. A call for clarity in reporting bleu scores. arXiv:1804.0877, 2018.
Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint /oXi::/608.0589 2016.
Abigail See, Minh-Thang Luong, and Christopher D Manning. Compression of neural machine translation models via pruming. arXiv preprint 1rXiv:1606.09274 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: /409. 1556, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pp- 1631-1642, 2013.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Compressing recurrent neural network with tensor train. arXiv preprint arXiv: 08052, 2017.
Ehsan Variani, Ananda Theertha Suresh, and Mitchel Weintraub. WEST: Word Encoded Sequence Transducers. arXiv preprint arXiv: /8/ 084 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 1998-6008, 2017.
10

---

Under review conference paper at ICLR 2020
Wenqi Wang. Yifan Sun, Brian Eriksson, Wenlin Wang, and Vaneet Aggarwal. Wide compression: Tensor ring nets. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 9329-9338, 2018
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint orXiv: 10430, 2019.
Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, and Hongkai Xiong. Deep neural network compression with single and multiple level quantization. arXiv preprint arXiv: 803.03289, 2018.
Yinchong Yang. Denis Krompass, and Volker Tresp. Vensor-train recurrent neural networks for video classification. arXiv preprint arXiv:1707.01/ 2017.
Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using tensor-train RNNs. arXiv preprint
2017.
Qibin Zhao, Guoxu Zhou, Shengli Xie, Liqing Zhang, and Andrzej Cichocki. Tensor ring decomposi- tion. arXiv preprint arXiv: 05535, 2016.
11

---

Under review conference paper at ICLR 2020
A MULTIINDEX CONSTRUCTION
Algorithm The algorithm implementing the bijection I(i) as described in Section 3.3.
Require: / - vocabulary size, { - an arbitrary factorization of I, i index of the target word in vocabulary.
Returns: I(i) = (11..... iN) N-dimensional index.
Initialize: L = 1, 11, I1.2..... 11 ... .N-1
for N to do
i < fl rl
c mod [k]
end for
Algorithm 2 The algorithm implementing the bijection (iz,..., ,N)) > i, inverse to I(i).
Require:
vocabulary size, { arbitrary factorization of I, (i1...., N-
dimensional index.
Returns:
index of the target word in vocabulary Initialize: L = {1, I1, I I2...., I1 IN-1
for to N do
i i + ig [[]]
end for
B TENSOR RING EMBEDDING
Tensor Ring (TR) decomposition is generalization TT-decomposition where the first and the last cores are 3-dimensional tensors which corresponds to Ru = R2 > Formally, a tensor X is said to be represented in the TR format (Zhao et al., 2016) if each element of X can be computed as:
Ra
R
RN-1
X (i1, ,2,
i2)
Σ
(ro, é1, (1) (1,,2,22) (FN-1,, ro).
10=1n
TN-1=1
Similar to TT, we can define TR-matrix (see Figure 3) and corresponding TR-embedding layer.
1: J2
Ri
Rs R
R1
Reshaping
TR decomposition
R:
1
1 1J
Is J3
Js
Ia J2
R₁
R.
I X J3
Figure 3: Construction the TR-matrix from the standard embedding matrix. Blue color depicts how the single element in the initial matrix is transformed into the product of the highlighted matrices.
Table 5 shows the performance of different NMT models with both embedding and softmax layers replaced by either TT TR factorizations. To achieve the same compression factor as the correspond- ing TT models, TR models should have smaller ranks which negatively affects their performance. Furthermore, TR is more computationally heavy.
C COMPLETE LIST OF HYPERPARAMETER
12

---

Under review conference paper at ICLR 2020
Table 5: NMT, Transformer-big WMT English-to-German dataset. Both case-sensitive
tokenized BLEU and de-tokenized SacreBLEU (Post, 2018) on newstest2014 are reported.
TT rank
Token BLEU
Sacre BLEU
Emb compr.
Total params
Iter
Model
Embedding shape
time
Big
32768 x 1024
29.58
28.84
210M
1.14
Big+TTI
(32, 02,32,332 32)
(8, 8, 16)
64
29.17
28.53
15.3
179M
1.23
Big+TT2
(32, 32, 32)
(8,8, 16)
48
28.53
27.97
26.8
178M
1.22
Big+TT3
(32, 32,
X (8, 16)
32
28.26
27.70
58.5
177M
1.22
Big+ TR1
(32, 32, 32)
X (8, 8, 16)
32
28.64
28.07
16
179M
1.28
Big+The TRR+
(32, 32, 32)
(8, 8, 16)
28.10 10
27.50
177M
1.23
Table 6: Full list of hyperparameters used to train Transformer-big for NMT.
Parameter
Value
Data cleaning
training sequence length in tokens min training sequence length in tokens source / target difference in tokens max source / target ratio
128
25
2.5
Model
vocabulary size, |V]
32768
hidden size, dmodel
1024
intermediate FF layer size, du number of attention heads, h number of layers in encoder decoder
4096
16
Optimization
optimizer
NovoGrad
learning rate
0.04
betas, (31,32) learning rate decay policy weight decay batch size in tokens
(0.95, 0.25)
cosine
0.0001
393216
number of training steps
80000
number of warmup steps
4000
Regularization
global dropout, Parop
0.2
label smoothing
0.1
Inference
beam search beam size
length penalty
0
max source target difference in tokens
50
13

---

Under review conference paper at ICLR 2020
Table 7: Full list hyperparameters used to train Transformer-XL for LM.
Parameter
Value
Model
vocabulary size, [V]
267735
hidden size, elmodel intermediate FF layer size, dr number of attention heads, 8
512
2048
number of layers softmax layer
16
adaptive
Optimization
optimizer
NovoGrad
learning rate
0.025
betas, (51, 822
(0.95, 0.25)
learning rate decay policy weight decay batch size in sequences target sequence length
cosine
0.0001
1024
128
memory sequence length
128
number training steps
150000
number of warmup steps
1500
Regularization
global dropout, Parop
0.15
Inference
batch size
4
target sequence length
128
memory sequence length
640
max positional encodings length
400
14


Reviews:
[{"review_id": "Syxic2OTYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a novel way of parametrizing embedding layers based on the Tensor Train (TT)\ndecomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. And this paper focuses on the input embedding layers. \n\nFor the experiments, the paper just compared methods using TT layer and normal embedding layer. There are many other methods that has been proposed to compress the embedding layers, it will be good to compare with one or two other methods, such as WEST or compression based on projection layers. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJe4_DGZ9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use TensorTrain representation to transform discrete tokens/symbols to its vector representation.\nSince neural networks can only work with numerical numbers, in many NLP tasks, where the raw inputs are in the discrete token/symbol format, the popular technique is to use \"embedding\" matrices to find a vector representation of those inputs. \n\nAs the authors point out, the embedding matrices usually require huge number of parameters, since it assigns one vector for each input token for one embedding vector, but to attain a competitive performance in the real world applications, we need to use large number of embedding vectors, which results in a large number of parameters in the neural networks.\n\nThe paper assumes that those embedding matrices can be compressed by assuming that the low-rank property of embedding matrices. I think this is a valid assumption in many cases, and the paper shows the performance degradation according to this assumption is relatively small compared to the gain, a dramatically reduced size of parameters in the embedding stage, is substantial.\n\nI think the paper is well written and proposes a new direction to find a memory efficient representation of symbols. I am not sure the current initialization techniques, nor the training method in the paper are the right way to train a tensor train \"embedding\" but I expect that the authors would perform the follow up work on those topics.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklnwAZF_r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a low-rank tensor decomposition model (Tensor Train-TT [Oseledets et al, 2011]) to parameterize the embedding matrix in Natural Language Processing (NLP). It shows that TT allows for a compression of the network and sometimes even a slight increase of test accuracy. The paper is well written and easy to follow.\nI found the idea as a natural consequence of many recent papers proposing tensor decomposition to parameterize deep learning networks. However, I think this is the first time that the concept has been applied to learning an embedding matrix, which is an important problem in the field.\nThe authors reported several experimental results on different tasks and datasets for NLP such as: Sentiment Analysis, Neural Machine Translation and Language Modeling; and an application to the click through rate prediction problem.\nI think the paper is of limited novelty but includes interesting experimental results that helps to better understand the potential and limitations of tensor decompositions in deep learning architectures.\nBelow, I summarize the issues I found, and I would like the authors to address them in their responses:\nMajor issues:\n-\tIn Page 4 (and Appendix B), the authors show a comparison with Tensor Ring (TR) and conclude that TT marginally outperforms TR in terms of BLEU measure for a fixed number of parameters in both models. I found this comparison incomplete, weak and misleading because of the following reasons:\no\tTR is a more general model including TT as a particular case when the first and last ranks are equal to one (Zhao et al, 2016). In fact, in this experiment, the authors chose all intermediate ranks set at the same value R with the first/last ranks set to 1 and R for TT and TR, respectively, which is not a fair comparison. Shown results suggest that first/last ranks contain less information than intermediate ranks, but it would be necessary to explore other combinations of rank values without keeping them constant to explore the generalization power of the TR model including TT as a particular case.\no\tThe authors compares TT and TR only in case of the NMT, Transformer-big on WMT‘14 English-to-German dataset, where the results are not good for TT and TR. It is noted that baseline model (Big) attains a Sacre BLEU = 28.84, TT1 = 28.53 and TR1 = 28.07 and the compression rate is only 210/179=1.17 for TT1 and TR1 and the Iteration time is larger than in the baseline model. In this case, there is no a clear advantage of using TT or TR.\nIn my opinion, to improve the paper, I think the authors could:\no\tTo avoid the sentence “In our experiments, however, the resulting TR embeddings performed slightly worse than TT–embeddings with the same number of parameters” unless more conclusive and exhaustive experiments are performed comparing TT and TR.\no\tTo add some comparison results between TT and TR for the rest of datasets such as Sentiment Analysis, Language Modeling and the click through rate prediction problem. \no\tHighlight that TT is a particular case of TR so considering the first and last ranks equal to one reduce the number of parameters but can affect the generalization power of the model.\n-\tThe approach of the paper is mostly intuitive. A theoretical result about why the low rank TT is able to catch the useful information of an optimal or suboptimal embedding matrix is missing.\nMinor issues:\n-\tIn last paragraph of section 3.2: The number of parameters is computed on the 3D tensor cores only. I think the size of the first/last 2D cores should be added. Please revise the equation.\n-\tThe pseudocode for the mapping one index to multiple indices is trivial and could be avoided. If it is kept, I think the reverse operation should be also included, i.e. how to map multiple indices i1, …., iN to one index i.\n-\tThe discussion and Figure 2 about the Gaussianity of the values in the higher order tensor based on Gaussian core tensors is not relevant. Maybe, the authors should better motivate why it is important to highlight that the distribution tends to a Gaussian density for increasing ranks.\n-\tIn page 5, it is mentioned that “factors should be as close to each other as possible” but there is no a justification for it. Could you give some theoretical insight on why it is important to obtain uniform distribution of matrix size?\n-\tSection 4.1, reference to the Stanford sentiment treebank (SST) is missing.\n\nOn Nov 16th: I am satisfied with the responses provided by the Authors who made few changes to solve some identified minor issues. Thanks for taking the review report into account. I have raised the rating. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been reviewed by three reviewers and received scores: 6/3/8. While two reviewers were reasonably positive, they also did not provide a very compelling reviews (e.g. one rev. just reiterated the rationale behind tensor model compression and the other admitted the paper is of limited novelty). Perhaps the shortest review (and perhaps the most telling) prompts authors to the fact that the model compression with tensor decompositions is quite common in the literature these days. One example could be T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor by Kossaifi et al. Very likely the authors will find many more recent developments on model compression with/without tensor decomp. For a good paper in this topic, authors should carefully consider various tensor factorizations (Tucker, TT, tensor rings, t-product and many more) and consider theoretical contributions and guarantees. Taking into account all pros and cons, this submissions falls marginally short of the ICLR 2020 threshold but the authors are encouraged to work on further developments.
Decision: Reject