IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x

1

PAPER

Recurrent Neural Network Compression based on
Low-Rank Tensor Representation
Andros TJANDRA†,††a) , Nonmember, Sakriani SAKTI†,††b) , and Satoshi NAKAMURA†,††c) ,SUMMARY Recurrent Neural Network (RNN) has achieved manystate-of-the-art performances on various complex tasks related to the temporal and sequential data. But most of these RNNs require much computational power and a huge number of parameters for both training andinference stage. Several tensor decomposition methods are included such as
CANDECOMP/PARAFAC (CP), Tucker decomposition and Tensor Train(TT) to re-parameterize the Gated Recurrent Unit (GRU) RNN. First, weevaluate all tensor-based RNNs performance on sequence modeling tasks
with a various number of parameters. Based on our experiment results,
TT-GRU achieved the best results in a various number of parameters compared to other decomposition methods. Later, we evaluate our proposed
TT-GRU with speech recognition task. We compressed the bidirectional
GRU layers inside DeepSpeech2 architecture. Based on our experimentresult, our proposed TT-format GRU are able to preserve the performance
while reducing the number of GRU parameters significantly compared tothe uncompressed GRU.
key words: recurrent neural network, model compression, tensor decomposition, deep learning

1.

Introduction

Modeling and predicting temporal sequential data are majortask in the machine learning field. In recent years, recurrent
neural network (RNN) has been a prominent choice for thesetasks. Despite it has been studied for about two decades [1,2],
the renaissance just arrived recently thanks to the significant
improvement of current computational power and the amountof available data. There are many state-of-the-arts in several
applications such as speech recognition [3, 4] and machinetranslation [5–7] had been achieved with RNN models.
Despite the fact that RNN produced impressive performance, most RNN models are computationally expensive
and have a huge number of parameters. Inside an RNN, theoutput are calculated by linear projection between matrices
and vectors, followed by nonlinear transformations, we needmultiple high-dimensional dense matrices as parameters. Inbetween two time-steps, we need to apply linear projection
between our dense matrix with high-dimensional input andprevious hidden states. Especially for state-of-the-art models on speech recognition [4] and machine translation [5],Manuscript received January 1, 2015.
Manuscript revised January 1, 2015.
† The author are with the Augmented Human Communication
Lab, Nara Institute of Science and Technology, Ikoma-shi, 6300192 Japan.
† The authors are with the RIKEN, Center for Advanced Intelligence Project AIP, Ikoma-shi, 630–0192 Japan.
a) E-mail: andros.tjandra.ai6@is.naist.jpb) E-mail: ssakti@is.naist.jpc) E-mail: s-nakamura@is.naist.jp
DOI: 10.1587/transinf.E0.D.1

such huge models can only be implemented in high-end
cluster environments because they need massive computation power and millions of parameters. These limitations
restrict the creation of efficient RNN models that are fastenough for massive real-time inference or small enough to
be implemented in low-end devices like mobile phones [8]or embedded systems with limited memory.
There is a trade-off between high accuracy model andhuge resources requirement with fast and smaller model withlow computational and memory costs. Some researchers
have done notable work to minimize the accuracy loss andmaximize the model efficiency. Hinton et al. [9] and Ba
et al. [10] successfully compressed a large deep neural network into a smaller neural network by training the latter on
the transformed softmax outputs from the former. Distillingknowledge from larger neural networks has also been successfully applied to recurrent neural network architectureby [11]. Denil et al. [12] utilized low-rank matrix decomposition of the weight matrices. A recent study by Novikov etal. [13] replaced the dense weight matrices with Tensor Train(TT) format [14] inside convolutional neural network (CNN)
model. With the TT-format, they significantly compress thenumber of parameters and kept the model accuracy degradation to a minimum. However, to the best of our knowledge,
no study has focused on compressing more complex neuralnetworks such as RNNs with tensor-based representation.
In this paper, we utilized several tensor decomposition
methods including CP-decomposition, Tucker decomposition and TT-decomposition for compressing RNN parameters † . We represent GRU RNN weight matrices with thesetensor decomposition methods. First, we conduct extensive
experiments on sequence modeling with a polyphonic music
dataset. We compare the performances of uncompressed
GRU model and three different tensor-based compressed
RNN models: CP-GRU, Tucker-GRU and TT-GRU [15] onvarious number of parameters. From our experiment results, we conclude that TT-GRU achieved the best resultin various number of parameters compared to other tensordecomposition method. Later, we conducted another experiment on speech recognition task. We modified a popular
end-to-end speech recognition model (DeepSpeech2 [4]) by
replacing the GRU with TT-GRU to reduce the number of parameters. We achieve high-compression ratio and maintain† Parts of this work were previously presented in [15–17]. In
this paper we summarized those works and provided a more detailed
description and comparison between all tensor-based RNNs.

Copyright © 200x The Institute of Electronics, Information and Communication Engineers

IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x2

the recognition accuracy from the compressed model.
In Section 2, we briefly review about RNN architectures
and their formulations. In Section 3, we explain about tensor
decomposition methods. In Section 4, we describe the details of our proposed tensor-based RNN model and how wetensorized the weight parameters inside RNN. In Section 5,
we describe about polyphonic music modeling task, including the dataset, model settings and experimental results. In
Section 6, we describe about speech recognition task, including the dataset, model description and experimental result.
We present related works in Section 7. Finally, we conclude
our result in Section 8.
2.

Recurrent Neural Network

2.1

Simple Recurrent Neural Network

An RNN is a kind of neural network architecture that models sequential and temporal dependencies [18]. Typically,
we define input sequence x = (x1 , ..., xT ), hidden vector sequence h = (h1 , ..., hT ) and output vector sequencey = (y1 , ..., yT ). As illustrated in Figure 1, a simple RNN attime t is can be formulated as:
ht = f (W xh xt + Whh ht−1 + bh )yt = g(Why ht + by ).

(1)
(2)

where W xh represents the weight parameters between the
input and hidden layer, Whh represents the weight parameters between the hidden and hidden layer, Why represents
the weight parameters between the hidden and output layer,
and bh and by represent bias vectors for the hidden and output layers. Functions f (·) and g(·) are nonlinear activation
functions, such as sigmoid or tanh.

simple RNNs. For example, Le et al. [21] replaced the activation function that causes the vanishing gradient with arectifier linear (ReLU) function. With an unbounded activation function and identity weight initialization, they optimized a simple RNN for long-term dependency modeling.
Martens et al. [22] used a second-order Hessian-free (HF)
optimization method rather than the first-order method suchas gradient descent. However, estimation of the second-order
gradient requires extra computational steps. Modifying theinternal structure from RNN by introducing gating mechanism also helps RNNs solve the vanishing and exploding
gradient problems. The additional gating layers control theinformation flow from the previous states and the current input [2]. Several versions of gated RNNs have been designed
to overcome the weakness of simple RNNs by introducing
gating units, such as Long-Short Term Memory (LSTM)
RNN and GRU RNN. In the following subsections, we explain both in more detail.
2.2.1

Long-Short Term Memory RNN

The LSTM RNN was proposed by Hochreiter et al. [2].
LSTM is a gated RNN with three gating layers and memory
cells, utilizes the gating layers to control the current memory
states by retaining the valuable information and forgetting theunneeded information. The memory cells store the internal
information across time steps. As illustrated in Figure 2,
the LSTM hidden layer values at time t are defined by thefollowing equations [23]:it = σ(W xi xt + Whi ht−1 + Wci ct−1 + bi )ft = σ(W x f xt + Wh f ht−1 + Wc f ct−1 + b f )ct = ft ct−1 + it tanh(W xc xt + Whc ht−1 + bc )ot = σ(W xo xt + Who ht−1 + Wco ct + bo )ht = ot tanh(ct )where σ(·) is sigmoid activation function and it , ft , ot andct are respectively the input gates, the forget gates, the output gates and the memory cells. The input gates retain thecandidate memory cell values that are useful for the current
memory cell and the forget gates retain the previous memory
cell values that are useful for the current memory cell. The
output gates retain the memory cell values that are useful forthe output and the next time-step hidden layer computation.

Figure 1

2.2

Recurrent Neural Network

Gated Recurrent Neural Network

Simple RNNs cannot easily be used for modeling datasetswith long sequences and long-term dependency because thegradient can easily vanish or explode [19, 20]. This problem is caused by the effect of bounded activation functions
and their derivatives. Therefore, training a simple RNN ismore complicated than training a feedforward neural network. Some researches addressed the difficulties of training

2.2.2

Gated Recurrent Unit RNN

The GRU RNN was proposed by Cho et al. [24] as an alternative to LSTM. There are several key differences between GRU and LSTM. First, a GRU does not have memory
cells [25]. Second, instead of three gating layers, it only has
two: reset gates and update gates. As illustrated in Figure 3,
the GRU hidden layer at time t is defined by the following
equations [24]:rt = σ(W xr xt + Whr ht−1 + br )zt = σ(W xz xt + Whz ht−1 + bz )

(3)
(4)

TJANDRA et al.: RECURRENT NEURAL NETWORK COMPRESSION BASED ON LOW-RANK TENSOR REPRESENTATION

3

parameters in the RNN. First, we define the tensorization process to transform the weight matrices inside the RNN model
into higher order tensors. Then, we describe three tensor decompositions method called as CANDECOMP/PARAFAC(CP) decomposition, Tucker decomposition and Tensor
Train (TT) decomposition.
3.1

Figure 2

Long Short Term Memory Unit.

h̃t = f (W xh xt + Whh (rt ht−1 ) + bh )ht = (1 − zt ) ht−1 + zt h̃t

(5)
(6)

where σ(·) is a sigmoid activation function, f (·) is a tanh
activation function, rt , zt are the reset and update gates, h̃t
is the candidate hidden layer values, and ht is the hidden
layer values at time-t. The reset gates control the previous
hidden layer values that are useful for the current candidate
hidden layer. The update gates decide whether to keep theprevious hidden layer values or replace the current hidden
layer values with the candidate hidden layer values. GRU
can match LSTM’s performance and its convergence speed
sometimes surpasses LSTM, despite having one fewer gating
layer [25].

Vector, Matrix and Tensor

Before we start to explain any further, we will define different
notations for vectors, matrices and tensors. Vector is an
one-dimensional array, matrix is a two-dimensional arrayand tensor is a higher-order multidimensional array. In thispaper, bold lower case letters (e.g., b) represent vectors,
bold upper case letters (e.g., W) represent matrices and boldcalligraphic upper case letters (e.g., W) represent tensors.
For representing the element inside vectors, matrices andtensors, we explicitly write the index in every dimension
without bold font. For example, b(i) is the i-th element invector b, W(p, q) is the element on p-th row and q-th column
from matrix W and W(i1 , .., id ) is the i1 , .., id -th index fromtensor W.
3.2

Tensor decomposition method

Tensor decomposition is a method for generalizing lowrank approximation from a multi-dimensional array. There
are several popular tensor decomposition methods, such as
Canonical polyadic (CP) decomposition, Tucker decomposition and Tensor Train decomposition. The factorization
format differs across different decomposition methods. In
this section, we explain briefly about CP-decomposition andTucker decomposition.
3.2.1

CP-decomposition

Canonical polyadic (CANDECOMP/ PARAFAC) decomposition [26–28] or usually referred to CP-decomposition factorizes a tensor into the sum of outer products of vectors.
Assume we have a 3rd-order tensor W ∈ Rm1 ×m2 ×m3 , we canapproximate it with CP-decomposition:
Figure 3

Gated Recurrent Unit

W≈

R
X

g1,r ⊗ g2,r ⊗ g3,r

(7)

r=1

In this section, we provided the formulation and the details for several RNNs. As we can see, most of the RNNs
consist of many dense matrices that represents a large number of weight parameters that are required to represent all of
the RNN models. In the next section, we present an alternative RNN model that significantly reduces the number ofparameters and simultaneously preserves the performance.

where ∀r ∈ [1..R], g1,r ∈ Rm1 , g2,r ∈ Rm2 , g3,r ∈ Rm3 , R ∈Z+ is the number of factors combinations (CP-rank) and ⊗denotes Kronecker product operation. Elementwise, we cancalculate the result by:

3.

In Figure 4, we provide an illustration for Eq. 7 in moredetails.

Tensor Decomposition

In this section, we explain our approaches to compress theW(x, y, z) ≈RX

g1,r (x) g2,r (y) g3,r (z)

(8)

r=1

IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x

4W(x, y, z) ≈

r1 X
r2
X

G1 (x, s1 )G2 (s1 , y, s2 )G3 (s2 , z)s1 =1 s2 =1

3.2.2

where x ∈ [1, .., m1 ], y ∈ [1, .., m2 ], z ∈ [1, .., m3 ] andG1 ∈ Rm1 ×r1 , G2 ∈ Rr1 ×m2 ×r2 , G3 ∈ Rr2 ×m3 as the TT-cores.
Figure 6 gives an illustration for Eq. 11.

CP-decomposition for 3rd-order tensor W

Figure 4(11)

Tucker decomposition

Tucker decomposition [28, 29] factorizes a tensor into a coretensor multiplied by a matrix along each mode. Assume wehave a 3rd-order tensor W ∈ Rm1 ×m2 ×m3 , we can approximate
it with Tucker decomposition:
W ≈ G0 ×1 G1 ×2 G2 ×3 G3

where G0 ∈ Rr1 ×r2 ×r3 is the core tensor, G1 ∈ Rm1 ×r1 ,G2 ∈ Rm2 ×r2 , G3 ∈ Rm3 ×r3 are the factor matrices and ×n is
the n-th mode product operator. The mode product between
a tensor G0 ∈ Rn1 ×n2 ×n3 and a matrix G1 ∈ Rm1 ×n1 is a tensor
Rm1 ×n2 ×n3 . By applying the mode products across all modes,
we can recover the original W tensor. Elementwise, we cancalculate the element from tensor W by:W(x, y, z) ≈

r3
r1 X
r2 X
X

G0 (s1 , s2 , s3 )s1 =1 s2 =1 s3 =1G1 (x, s1 ) G2 (y, s2 ) G3 (z, s3 )Figure 6

(9)

(10)

where x ∈ [1, .., m1 ], y ∈ [1, .., m2 ], z ∈ [1, .., m3 ]. Figure 5
gives an illustration for Eq. 9

Tensor Train decomposition for 3rd-order tensor W

To generalized TT-decomposition for d-dimensional
tensor, TT-decomposition equation can be described as:W( j1 , j2 , .., jd−1 , jd ) =G1 ( j1 ) · G2 (:, j2 , :) · ... · Gd−1 (:, jd−1 , :) · Gd (:, jd ).

(12)where jk ∈ [1, .., mk ]. For all vectors or matrices sliced fromtensor Gk , if Gk (:, jk , :) related to the same dimension k, theymust be represented with size rk−1 × rk , where r0 and rd
must be equal to 1 to retain the final matrix multiplication
result as a scalar. In TT-format, we define a sequence ofrank {rk }dk=0 and we call them TT-rank from tensor W. The
set of matrices Gk = {Gk (:, jk , :)}mjkk=1 where the matrices arespanned in the same index are called TT-core.
We can generalized the TT-format equation in detail byenumerating the index qk−1 ∈ {1, .., rk−1 } and qk ∈ {1, .., rk }in matrix Gk ( jk ) across all k ∈ {1, .., d}:W( j1 , j2 , .., jd−1 , jd ) =X
G1 (q0 , j1 , q1 ) ... Gd (qd−1 , jd , qd ). (13)q0 ,..,qd

Figure 5

3.2.3

Tucker decomposition for 3rd-order tensor W

Tensor Train decomposition

Tensor Train decomposition [14] factorizes a tensor into acollection of lower order tensors called as TT-cores. All TTcores are connected through matrix multiplications acrossall tensor order to calculate the element from original tensor.
Assume we have a 3rd-order tensor W ∈ Rm1 ×m2 ×m3 , we canapproximate the element at index x, y, z by:

By factoring the original tensor W into multiple TTcores {Gk }dk=1 , we can compress the number of elements
Qneeded to represent the original tensor size from dk=1 mk toPd
k=1 mk rk−1 rk .
Lastly, to create an intuitive example on how to represent tensor using set of TT-cores, we illustrate how torepresent a tensor W element at (1, 0, 3) with 3 TT-cores in
Figure 7.
4.

Proposed Tensor-based RNN

In this section, we describe our proposed approach tocompress RNN model using various tensor decomposition

TJANDRA et al.: RECURRENT NEURAL NETWORK COMPRESSION BASED ON LOW-RANK TENSOR REPRESENTATION+ B(fi (p))

5

(15)

by enumerating all columns q position with j1 , .., jd andfi (p) = [i1 (p), .., id (p)].
Based on the Eq. 15, we rewrite the linear layer equationdifferently depends on the the decomposition method:
Figure 7 Representing a tensor W element at (1, 0, 3) using 3 TT-cores
G1 , G2 and G3 . Blue shaded vectors or matrices are used for chain multiplication

method that we have described above. First, we will explain
about linear layer tensorization. After that, we rewrite the
RNN equation based on the from the matrices weight format
into the tensorized weight format.
4.1

Representing Linear Transformation on Tensorized
WeightMost of RNN equations are composed by multiplication between the input vector and their corresponding weight matrix:
y = Wx + b(14)

where W ∈ R M×N is the weight matrix, b ∈ R M is thebias vector and x ∈ RN is the input vector. Thus, mostof RNN parameters are used to represent the weight matrices. To reduce the number of parameters significantly, weneed to represent the weight matrices with the factorizationof higher-order tensor. First, we apply tensorization on theweight matrices. Tensorization is the process to transforma lower-order dimensional array into a higher-order dimensional array. In our case, we tensorize RNN weight matrices
into tensors. Given a weight matrix W ∈ R M×N , we can repm1 ×m2 ×..×md ×n1 ×n2 ×..×nd whereresentQthem as a tensor W
Qd ∈ Rd
M = k=1 mk and N = k=1 nk . For mapping each element
in matrix W to tensor W, we define one-to-one mapping between row-column and tensor index with bijective functions
fi : Z+ → Zd+ and f j : Z+ → Zd+ . Function fi transforms eachrow p ∈ {1, .., M} into fi (p) = [i1 (p), .., id (p)] and f j transforms each column q ∈ {1, .., N} into f j (q) = [ j1 (q), .., jd (q)].
Following this, we can access the value from matrix W(p, q)
in the tensor W with the index vectors generated by fi (p)and f j (q) with these bijective functions.
After we determine the shape of the weight tensor, we choose one of the tensor decomposition methods(e.g., CP-decomposition (Sec.3.2.1), Tucker decomposition(Sec.3.2.2) or Tensor Train (Sec.3.2.3)) to represent and reduce the number of parameters from the tensor W. In orderto represent matrix-vector products inside RNN equations,
we need to reshape the input vector x ∈ RN into a tensor
X ∈ Rn1 ×..×nd and the bias vector b ∈ R M into a tensor
B ∈ Rm1 ×..×md . Therefore, we can reformulate the Eq. 14 to
calculate y(p) elementwise with:
Y(fi (p)) =X
j1 ,.., jd
W fi (p), j1 , .., jd X( j1 , .., jd )1. CP-decomposition:
For CP-decomposition, we represent our tensor W withmultiple factors gmk,r , gnk,r where ∀k ∈ [1..d] ∀r ∈[1..R], (gmk,r ∈ Rmk , gnk,r ∈ Rnk ). From here, we replace Eq. 15 with: R d
X XY
Y(fi (p)) =gmk,r (ik (p))gnk,r ( jk ) X( j1 , .., jd )
j1 ,.., jd r=1 k=1+ B(fi (p)).
(16)
By using CP-decomposition for representing the weightmatrix W, we reduce
the number of parameters fromP
M × N into R ∗ ( dk=1 mk + nk ).
2. Tucker decomposition:
For Tucker decomposition, we represent our tensor W
with a tensor core G0 ∈ Rr1 ×...×rd ×rd+1 ×...×r2d where∀k ∈ [1..d], rk < mk and ∀k ∈ [1..d], rd+k < nkand multiple factor matrices GMk , GNk , where ∀k ∈[1..d], (GMk ∈ Rmk ×rk , GNk ∈ Rnk ×rd+k ). Generally,
the tensor core ranks r1 , r2 , .., rd are corresponding to
the row in tensor index and rd+1 , rd+2 , .., r2d are corresponding to the column in tensor index. From here, we
replace Eq. 15 with:,rd+1 ,..,r2d
X r1 ,..,rdXG0 (s1 , ..sd , sd+1 , .., s2d )Y(fi (p)) =
j1 ,.., jd s1 ,..sd ,sd+1 ,..,s2d
d
Y
k=1



GMk (ik (p), sk )GNk ( jk , sd+k ) X( j1 , .., jd )+ B(fi (p)).
(17)
By using Tucker decomposition for representing theweight matrix W,Pwe reduce the number of parameters
Q
from M × N into dk=1 (mk ∗ rk + nk ∗ rd+k ) + ( 2dk=1 rk ).
3. Tensor-train decomposition:
For Tensor-train decomposition, we represent our tensor W with multiple TT-cores Gm1 ..md and Gn1 ..nd .
TT-cores Gk ∈ Rrk−1 ×mk ×rk , ∀k ∈ [1..d] are corresponding to the row in tensor index and TT-cores
Gd+k ∈ Rrd+k−1 ×nk ×rd+k , ∀k ∈ [1..d] are corresponding
to the column in tensor index. rk−1 ..rd are the TT-ranks
for row TT-cores and rd+k−1 ..r2d are the TT-ranks forcolumn TT-cores. From here, we replace Eq. 15 with:IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x

6

Y(fi (p)) =X

G1 (:, i1 (p), :)..Gd (:, id (p), :)j1 ,.., jd

·Gd+1 (:, j1 , :)..G2d (:, jd , :) X( j1 , .., jd )+ B(fi (p)).
(18)
By using TT-decomposition for representing the weightmatrix W, P
we reduce the number of parameters from
M × N into dk=1 (rk−1 ∗ mk ∗ rk ) + (rd+k−1 ∗ nk ∗ r2d ) .
We can control the shape of TT-cores {Gk }2di=1 by choosing factor M as {mk }dk=1 and N as {nk }dk=1 as long as thenumber of factors is equal between M and N. We canalso define TT-rank and treat them as a hyper-parameter.
In general, if we use a smaller TT-rank, we will get more
efficient models but this action restricts our model tolearn more complex representation. If we use a larger
TT-rank, we get more flexibility to express our weightparameters but we sacrifice model efficiency.

4.2

j1 ,.., jd

Compressing Simple RNN

We represent a simple RNN in tensor-based format. From
Section 2.1, we focus our attention on two dense weightmatrices: (W xh , Whh ). Previously, we defined W xh ∈ R M×N
as input-to-hidden parameters and Whh ∈ R M×M as hiddento-hidden parameters and bh .
Q
First, we factorize matrix shape M into dk=1 mk andQd
N into k=1 nk . Next, we determine which tensor decomposition method to represent the weight matrices. We will
substitute weight matrix W xh with tensor W xh and Whh withtensor Whh . We define bijective functions fix and fih to access row p ∈ [1..M] from W xh and Whh in the tensor-based
representation. We rewrite our simple RNN formulation to
calculate ht in Eq.1:
X
atxh (p) =W xh (fix (p), [ j1 , .., jd ]) · Xt j1 , .., jd
j1 ,.., jdXahht (p) =ahrt (p) =X

Whr (fih (p), [ j1 , .., jd ]) · Ht−1 j1 , .., jd

j1 , .., jdj1 ,.., jdiatxh = atxh (1), .., atxh (M)
h
i
hh
hh
ahh
t = at (1), .., at (M)ht = f (atxh + ahh
t + bh ),where X is the tensor representation of input xt and Ht−1 isthe tensor representation of previous hidden states ht−1 .
Compressing GRU RNN

In this section, we apply tensor-based format to represent agated RNN. Among several RNN architectures with gating
mechanism, we choose GRU to be reformulated in tensorbased representation because it has less complex formulationand similar performance as LSTM. For the rest of this paper,

j1 ,.., jd
atxr = atxr (1), .., atxr (M)
h
i
hr
hr
ahr
t = at (1), .., at (M)rt = σ(atxr + ahr
t + br ).

(19)

Next, we rewrite the GRU formulation to calculate zt in Eq.4:
X
atxz (p) =W xz (fix (p), [ j1 , .., jd ]) · Xt j1 , .., jd
j1 ,.., jdahzt (p) =X

Whz (fih (p), [ j1 , .., jd ]) · Ht−1 j1 , .., jdj1 ,.., jd
atxz = atxz (1), .., atxz (M)
h
i
hz
hz
ahz
t = at (1), .., at (M)zt = σ(atxz + ahz
t + bz ).

Whh (fih (p), [ j1 , .., jd ]) · Ht−1

h4.3

we called Tucker-GRU if we used Tucker decomposition,
CP-GRU if we used CP-decomposition and TT-GRU if weused TT-decomposition to replace the weight matrices inside GRU. In Section 2.2.2, we focus on the following six
dense weight matrices: (W xr , Whr , W xz , Whz , W xh , and Whh ).
Weight matrices W xr , W xz , W xh ∈ R M×N are parameters for
projecting the input layer to the reset gate, the update gate,
the candidate hidden layer, and Whr , Whz , Whh ∈ R M×M are
respectively parameters for projecting previous hidden layer
into the reset gate, the update gate and candidate hidden
layer.
Q
We factorize matrix shape M into dk=1 mk and N intoQd
k=1 nk . All weight matrices (W xr , Whr , W xz , Whz , W xh ,Whh ) are substituted with tensors (W xr , Whr , W xz , Whz ,W xh , Whh ) in tensor-based weight representation. We define bijective function fix to access row p from W xr , W xz , W xh
and function fih to access row p from Whr , Whz , Whh in tensorbased weight representation. We rewrite the GRU formulation to calculate rt in Eq.3:
X
atxr (p) =W xr (fix (p), [ j1 , .., jd ]) · Xt j1 , .., jd(20)

Finally, we rewrite the GRU formulation to calculate h̃t in
Eq.5:
X

atxh (p) =W xh (fix (p), [ j1 , .., jd ]) · Xt j1 , .., jd
j1 ,.., jdahht (p) =X

Whh (fih (p), [ j1 , .., jd ]) ·j1 ,.., jd
Rt j1 , .., jd · Ht−1 j1 , .., jdihatxh = atxh (1), .., atxh (M)
h
i
hh
hh
ahh
t = at (1), .., at (M)h̃t = f (atxh + ahh
t + bh ).

(21)

After rt , zt and h̃t are calculated, we calculate ht on Eq.6 withstandard operations like element-wise sum and multiplication.

TJANDRA et al.: RECURRENT NEURAL NETWORK COMPRESSION BASED ON LOW-RANK TENSOR REPRESENTATIONIn practice, we could assign a different d for each weighttensor as long as the input data dimension can also be factorized into the d values. The choice of tensor decomposition
method, ranks, and factors shape are determined by the user
and treated as hyperparameter. However, to simplify our implementation we use the same d for both the input and hidden
factorization size.QFor TT-GRU, we Q
also use the same factorizations M = dk=1 mk and N = dk=1 nk for all weighttensors.
We do not substitute bias vector b into tensor B because
the number of bias parameters is insignificant compared to
the number of parameters in matrix W. In terms of performance, the element-wise sum operation for bias vector b
is also insignificant compared to the matrix multiplication
between a weight matrix and the input layer or the previous
hidden layer.
4.4

Tensor Core and Factors Initialization Trick

Because of the large number of recursive matrix multiplications, followed by some nonlinearity (e.g, sigmoid, tanh),the gradient from the hidden layer will diminish after several time-step [30]. Consequently, training recurrent neuralnetworks is much harder compared to standard feedforward
neural networks.
Even worse, we decompose the weight matrix into multiple smaller tensors or matrices, thus the number of multiplications needed for each calculation increases multiple
times. Therefore, we need a better initialization trick on thetensor cores and factors to help our model convergences inthe early training stage.
In this work, we follow Glorot et al. [31] by initializing
the weight matrix with a certain variance. We assume that
our original weight matrix W has a mean 0 and the variance
σ2W . We utilize the basic properties from a sum and a productvariance between two independent random variables.
Definition 4.1. Let X and Y be independent random variables with the mean 0, then the variance from the sum of X
and Y is Var(X + Y) = Var(X) + Var(Y)sσg =After we decided the target variance σ2w for our original

σ2w
R

(22)

and initialize gmk,r , gnk,r ∼ N(0, σ2g ).
Proof. Define 1) a dense matrix W and each element
are initialized with independent random variable w, 2)
all tensor factors gm, gn element are initialized withindependent random variable g. We approximate variable w by series of summation and multiplication fromelements of tensor factors random variable g (Eq. 16).

W(p, q) =R YdX

gmk,r (ik (p))gnk,r ( jk (q))(23)

r=1 k=1

We replace W(p, q) with random variable w andgmk,r (ik (p)), gnk,r ( jk (q)) with random variable g.
w=
w=

R Y
d
X

g∗g

r=1 k=1
R
X
2d

g

r=1
2d

w = RgApply variance for both sides.

Var(w) = Var Rg2d 
Var(w) = RVar g2dVar(w) = RVar g 2d

s
4d

(24)
(25)
(26)
(27)

(28)
(29)
(30)

σ2w = Var(w)σ2g = Var(g)σ2w= (σ2g )2d

(32)

σ2w
= σ2g
R

(34)

σ2w
= σg
R

(35)

R

s
2d

Definition 4.2. Let X and Y be independent random variables with the mean 0, then the variance from the product of
X and Y is Var(X ∗ Y) = Var(X) ∗ Var(Y)

4d

7

(31)

(33)

weight matrix, now we need to derive the proper initialization rules for the tensor core and factors. We calculate thevariance for tensor core and factors by observing the number of sum and product operations and utilize the varianceproperties from Def. 4.1 and 4.2. Here, we listed different
initialization strategies for each tensor decomposition methods:∗ Notes: g between different factors and positions are
not a same random variable (we just write as g for sakeof simplicity). Therefore, for the case where multiple
sum of g, we use Definition 4.1, not of Var(aX) =a2 Var(X). For the case of multiple multiplication of g,
we use Definition 4.2, not based on Chi-Square randomvariable and their variance calculation.

1. CP-decomposition:
For weight tensor W based on the CP-decomposition,
we can calculate σg as the standard deviation for all
factors gmk,r , gnk,r with:

2. Tucker decomposition:
For weight tensor W based on the Tucker decomposition, we can calculate σg as the standard deviation for

IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x8

the core tensor G0 and the factor matrices GMk , GNk
with:s(4d+2)
σ2w
σg =
(36)
Q2d
k=1 rkand initialize G0 , GMk , GNk ∼ N(0, σ2g ).
Proof. Define 1) a dense matrix W and each element are
initialized with independent random variable w, 2) all
tensor factors G0 , GM, GN element are initialized withindependent random variable g. We approximate variable w by series of summation and multiplication fromelements of tensor factors random variable g (Eq. 17).
r1 ,..,rdX,rd+1 ,..,r2dW(p, q) =G0 (s1 , ..sd , sd+1 , .., s2d )

s1 ,..,sd ,sd+1 ,.,s2ddY

GMk (ik (p), sk )GNk ( jk (q), sd+k )(37)

3. Tensor-train decomposition:
For weight tensor W based on the Tensor Train decomposition, we can calculate σg as the standard deviation
for all the TT-cores Gk with:sσ2w(4d−2)
(50)
σg =
Q2d
k=0 rkand initialize Gk ∼ N(0, σ2g ).
Proof. Define 1) a dense matrix W and each element
are initialized with independent random variable w, 2)
all tensor cores Gm1 ,..,md , Gn1 ,..,nd element are initialized with independent random variable g. We approximate variable w by series of summation and multiplication from elements of tensor factors random variable
g (Eq. 18).

k=1

We replace W(p, q) with random variable w and
G0 , GM, GN with random variable g.
w=
w=
w=

r1 ,..,rdX,rd+1 ,..,r2ds1 ,..,sd ,sd+1 ,.,s2dr1 ,..,rdX,rd+1 ,..,r2ds1 ,..,sd ,sd+1 ,.,s2dr1 ,..,rdX,rd+1 ,..,r2d

g

d
Y

g∗g

(38)

w=

g ∗ g2d(39)

We replace W(p, q) with random variable w and
Gm1 ,..,md , Gn1 ,..,nd with random variable g.
w=

2d
Y

g2d+1

(40)
(41)

k=1

Apply variance for both sides.
 2d

Y

2d+1


Var(w) = Var 
rk g

2d
Y



Apply variance for both sides.

 2d

Y
(2d−1)

rk g
Var(w) = Var 

rk Var g

Q2d

k=1 rk

s
2d+1

σ2w
Q2d

k=1 rk

s
4d+2

σ2w
Q2d

k=1 rk

2d+1



(54)

k=0

Var(w) =2dYrk Var g(2d−1)(55)


rk Var g (2d−1)

(56)

k=0

(42)
Var(w) =

2d
Y
k=0

(43)

k=1

σ2w = Var(w)

(57)

2d
Y

σ2g = Var(g)(58)

rk Var(g)2d+1

k=1
2
σw = Var(w)σ2g = Var(g)

σ2w

(52)
(53)

k=1

Var(w) =rk g(2d−1)

k=0

rk g2d+1

Var(w) =

(51)

k=1

s1 ,..,sd ,sd+1 ,.,s2d2dYW(p, q) = G1 (:, i1 (p), :)Gd (:, id (p), :)Gd+1 (:, j1 (q), :)G2d (:, jd (q), :)= (σ2g )2d+1
= σ2g

(44)

σ2w =

(45)

σ2w =

(47)

(49)

(59)

2d
Y

rk σ4d−2
g

(60)

k=0

σ2w
Q2d

= σ4d−2
g

(61)

σ2w
Q2d

= σg

(62)

k=0 rk

s
4d−2

= σg

rk σg2(2d−1)

k=0

(46)

(48)

2d
Y

k=0 rkTJANDRA et al.: RECURRENT NEURAL NETWORK COMPRESSION BASED ON LOW-RANK TENSOR REPRESENTATION

By choosing a good initialization, our neural network
will converge faster and obtain better local minima. Basedon our preliminary experiments, we get better starting loss at
the first several epochs compared to the randomly initialized
model with the same σk on Gaussian distribution for all set
of factors or cores.
5.

Experiments 1: Polyphonic Music ModelingIn this section, we describe our dataset and all model configurations. We performed experiments with three different
tensor-decompositions (CP decomposition, Tucker decomposition and TT decomposition) to compress our GRU andalso the baseline GRU. In the end, we report our experimentresults and compare various tensor decomposition methodwith different settings and number of parameters. Our
codes are available at https://github.com/androstj/
tensor_rnn.
5.1

DatasetWe evaluated our models with sequential modeling tasks.
We used a polyphonic music dataset [32] which contains 4
different datasets† : Nottingham, MuseData, PianoMidi and
JSB Chorales. For each active note in all time-step, we set
the value as 1, otherwise 0. Each dataset consists of at least
7 hours of polyphonic music and the total is ± 67 hours.
In Figure 8, we visualize the dataset and the task for thisexperiment.

5.2

9Models

We evaluate several models in this paper: GRU-RNN (nocompression), CP-GRU (weight compression via CP decomposition), Tucker-GRU (weight compression via Tucker decomposition), TT-GRU [15] (compressed weight with TTdecomposition). For each timestep, the input and output
targets are vectors of 88 binary value. The input vector is
projected by a linear layer with 256 hidden units, followed by
LeakyReLU [33] activation function. For the RNN model
configurations, we enumerate all the details in the following
list:
1. GRU• Input size (N): 256• Hidden size (M): 512
2. Tensor-based GRU• Input size (N): 256• Tensor input shape (n1..4 ): 4 × 4 × 4 × 4• Hidden size (M): 512• Tensor hidden shape (m1..4 ): 8 × 4 × 4 × 4
a. CP-GRU• CP-Rank (R): [10, 30, 50, 80, 110]b. Tucker-GRU• Core (G0 ) shape:– (2 × 2 × 2 × 2) × (2 × 2 × 2 × 2)
– (2 × 3 × 2 × 3) × (2 × 3 × 2 × 3)
– (2 × 3 × 2 × 4) × (2 × 3 × 2 × 4)
– (2 × 4 × 2 × 4) × (2 × 4 × 2 × 4)
– (2 × 3 × 3 × 4) × (2 × 3 × 3 × 4)
c. TT-GRU• TT-ranks:– (1 × 3 × 3 × 3 × 1)
– (1 × 5 × 5 × 5 × 1)
– (1 × 7 × 7 × 7 × 1)
– (1 × 9 × 9 × 9 × 1)
– (1 × 9 × 9 × 9 × 1)

Figure 8 A simple illustration for polyphonic music dataset. For eachtime-step, we have a vector of binary number. The colored circle denotes
the active note and the blank circle denotes the non-active note. The main
objective in this task is to predict which notes will be active at time-step t
given the information from previous active notes at time-step t − 1, t − 2, ...,
etc.
† Dataset are downloaded from:http://www-etud.iro.
umontreal.ca/~boulanni/icml2012

In this task, the training criterion is to minimize the negative log-likelihood (NLL). In evaluation, we measured two
different scores: NLL and accuracy (ACC). For calculating
the accuracy, we follow Bay et al. [34] formulation:PT
t=1 T P(t)ACC = PT(63)
t=1 T P(t) + FP(t) + FN(t)where T P(t), FP(t), FN(t) is the true positive, false positiveand false negative at time-t. We only used true positive (TP),false positive (FP), false negative (FN) and ignored the true
negative (TN) because most of the notes were turned off or

IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x

10

zero in the dataset.
For training models, we use Adam [35] algorithm forour optimizer. To stabilize our training process, we clip our
gradient when the norm ||∇w|| > 5. For fair comparisons,we performed a grid search over learning rates (1e − 2, 5e −3, 1e − 3) and dropout probabilities (0.2, 0.3, 0.4, 0.5). The
best model based on loss in validation set will be used forthe test set evaluation.
5.3

Result and Discussion

Figure 11 NLL comparison between TT-GRU, Tucker-GRU, and CPGRU on PianoMidi test set

Figure 9 NLL comparison between TT-GRU, Tucker-GRU, and CP-GRU
on Nottingham test set

Figure 12 NLL comparison between TT-GRU, Tucker-GRU, and CPGRU on MuseData test set

similar number of parameters. In some datasets (e.g., PianoMidi, MuseData, Nottingham), CP-GRU has better resultscompared to Tucker-GRU and achieves similar performance(albeit slightly worse) as TT-GRU when the number of parameters are greater than 6000. Overall, TT-GRU performed
the best in most of the settings. Therefore, for the next experiment, we will apply TT-GRU as our main compression
method.
6.
Figure 10 NLL comparison between TT-GRU, Tucker-GRU, and CPGRU on JSB Chorales test set

We report results of our experiments in Table. 1. For
the baseline model, we choose standard GRU-RNN without
any compression on the weight matrices. For the comparison between compressed models (CP-GRU, Tucker-GRU and
TT-GRU), we run each model with 5 different configurationsand varied the number of parameters ranged from 2232 up to12184. In Figure 9-12, we plot the negative log-likelihood(NLL) score corresponding to the number of parameters foreach model. From our results, we observe that TT-GRU performed better than Tucker-GRU in every experiments with

Experiment II: Speech Recognition

In this task, we will evaluate our proposed TT-GRU modelon speech recognition task. We describe our dataset forspeech recognition task and our end-to-end ASR model forthis task. In the end, we report our experiment results bycomparing the performance of uncompressed model with
TT-GRU compressed model.
6.1

Dataset

For the dataset, we used English speech corpus LibriSpeech
corpus [36] as a task to evaluate our proposed model model.
Due to time and resource constraints, we only used the smallest “train-clean-100” subset for training data, “dev-clean”

TJANDRA et al.: RECURRENT NEURAL NETWORK COMPRESSION BASED ON LOW-RANK TENSOR REPRESENTATION

11

Table 1 Comparison between all models and their configurations based on the number of parameters,
negative log-likelihood and accuracy of polyphonic test set
Model

ConfigGRU
IN:256
OUT:512CP-GRU
IN: 4,4,4,4
OUT: 8,4,4,4

TUCKER-GRU
IN: 4,4,4,4
OUT: 8,4,4,4

TT-GRU
IN: 4,4,4,4
OUT: 8,4,4,4

Table 2

Subsettrain-clean-100
dev-clean
test-clean

Rank
10
30
50
80
110
Cores
2,2,2,2
2,3,2,3
2,3,2,42,4,2,4
2,3,3,4TT-rank
1,3,3,3,1
1,5,5,5,1
1,7,7,7,1
1,9,9,9,1
1,11,11,11,1

ParamNottingham
NLL
ACC

DatasetJSB
PianoMidi
NLL
ACC
NLL
ACC

MuseData
NLL
ACC

1181184

3.369

71.1

8.32

30.24

7.53

27.19

7.12

36.30

2456
4296
6136
8896
11656

3.79
3.48
3.46
3.43
3.34

67.51
69.85
69.56
69.73
70.42

8.60
8.49
8.40
8.41
8.41

27.29
28.33
28.47
27.88
29.45

8.15
7.68
7.66
7.61
7.60

19.03
25.03
26.18
28.28
27.36

7.87
7.27
7.23
7.19
7.18

27.32
36.19
36.34
36.57
36.89

2232
4360
6408
10008
12184

3.71
3.64
3.55
3.52
3.41

68.30
68.63
69.10
69.18
70.23

8.57
8.48
8.44
8.41
8.43

27.28
28.10
28.06
27.70
29.03

7.98
7.75
7.73
7.75
7.69

20.79
24.92
25.66
24.46
25.26

7.81
7.38
7.69
7.38
7.43

29.94
34.20
32.50
35.58
33.63

2688
4096
6016
8448
11392

3.49
3.45
3.40
3.35
3.38

69.49
69.81
70.72
70.82
70.51

8.37
8.38
8.37
8.36
8.37

28.41
28.86
28.83
29.32
29.55

7.60
7.58
7.57
7.58
7.58

26.95
27.46
27.58
27.62
28.07

7.49
7.50
7.23
7.20
7.16

34.99
33.37
36.53
37.81
36.54

Librispeech dataset information

hours
100
5
5

speakers
251
40
40

per-spk minutes
25
10
10

subset for validation data, and “test-clean” subset for test
data as shown in Table 2. The speech utterances were segmented into multiple frames with a 25-ms window size and a
10-ms step size. Then we extracted 23-dimension filter bankfeatures using Kaldi’s feature extractor [37] and normalized
them to have zero mean and unit variance.
6.2

Model

First, we will briefly introduce the ASR system that we use inthis experiment. Our baseline ASR model is based on “DeepSpeech 2” architecture [38]. DeepSpeech2 model consists
multiple different type of layers, including convolutional
layer, recurrent bidirectional GRU layer, and fully connected
layer. Given an input speech X (i) and transcription Y (i) sampled from the training set D = {(X (1) , Y (1) ), (X (2) , Y (2) ), ...}, asingle speech utterance is represented as a matrix X (i) ∈ RS ×D
where T is the length of speech utterance and D is the feature
dimension. A transcription output Y (i) = [y1 , y2 , ..., yS ] is asequence of phoneme or grapheme plus a blank characterwith length T .
We illustrated the model architecture on the left side of
Figure 13. First, the speech features X are projected by one ormore 1D convolution (convolution over the time dimension)and transformed by a nonlinear activation function ReLU:
hls = f (Wl ~ hl−1s−c:s+c )(64)

where Wl is a convolution filter weight at layer-l, c is thecontext window size and f (x) = max(0, x) is a rectifier linear
unit (ReLU) activation function. After that, the output fromconvolutional layers hl1:S is fed into multiple bidirectional
GRU layers:
→
−l
−−l→
ht = GRU(hl−1t , ht−1 )←−←l−−hlt = GRU(hl−1t , ht+1 ).

(65)
(66)

→
−
The result from left-to-right GRU hlt is combined with the
←
−
→
− ←
−
hlt by sum between two hidden states hlt = hlt + hlt . Next, theresult from bidirectional GRU are fed into one or more fully
connected layers, followed by nonlinear activation function
ReLU:
hlt = f (W l hl−1
+ bl ).
t(67)

The last output from our model htL are used to represent
the label probability:
exp(wcL ∗ htL−1 + bc ).
P(yt = c|X) = P
L
L−1 + b )kk exp(wk ∗ ht

(68)

To train this model, we use Connectionist Temporal Classification (CTC) loss function [39].
We perform the TT-format representation on the bidirectional GRU layers in the middle of DeepSpeech2 architecture. In Figure. 13, we mark the layer that we change from
GRU to TT-GRU.
The setting of the model parameters of the GRU baseline is based on the paper of Deep Speech 2 [3], and theconfiguration of the parameters of TT-GRU is determinedbased on the tensor decomposition. For baseline GRU, weIEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x

12

and thus a character error rate (CER) was used for the evaluation function. The best system of our proposed model could
drastically reduce the parameters from 13 millions to 37,000.
Overall, this reduction was about 99% in the converted GRU
layer and about 60% in the entire model. This reveals that theperformance could be maintained while reducing the numberof parameters. Importantly, unlike several published systems
using these benchmarks, our proposed system does not involve a language model. Therefore, the results reported in
the paper could not reach state-of-the-art performance. Nevertheless, the results are still convincing as evidence of the
proposed framework’s effectiveness.
7.

Figure 13

We replace multiple layers of bidirectional GRU with TT-GRU

Table 3 Parameters, Compression Rate, Validation CER and Test CER
on Librispeech dataset

Model
GRU-H1510

Params13MCompr.
100

Val CER
20.03%

Test CER
20.62%TTGRU
H4x8x6x8-R3
H4x8x6x8-R5H4x8x6x8-R7

11K
22K
37K

0.08
0.16
0.27

27.57%
23.76%
22.68%

27.21%
23.40%
23.73%

used the SGD optimizer. However, our TT-GRU could notconverge with SGD, and thus we used the Adam algorithm tooptimize the TT-GRU model parameters. We evaluate ASR
performance by calculating the normalized edit-distance between the generated transcription and the ground truth. Here,
we use character error-rate (CER) because our output token
are set of alphabet letters (a-z) and space character.
6.3

Result and Discussion

Table 3 shows the performance of the proposed TT-GRU
in comparison with the baseline (uncompressed) GRU. The
baseline model is a GRU with 1510 hidden units. Our proposed model has a 4x8x6x8 output shape and TT-GRU of
tensor train rank (3, 5, 7). No language model was applied,

Related Work

Compressing parameters on neural network architecture has
become an interesting topic over the past several years dueto the increased complexity of neural networks. The numberof parameters and processing times has also grown tremendously along with their performance. A number of researchers comes up with many different ways to tackle thisproblem.
Ba et al. [10] and Hinton et al. [9] “distilled" the knowledge from a deep neural network into a shallow neural network. First, they trained a state-of-the-art model with a deepand complex neural network using the original dataset andhard label as the target. After that, they reused the trained
deep neural network by extracting output from the softmax
layer and used them as the output target for a shallow neuralnetwork. By training the shallow network with a soft target,they achieved a better performance than the model trainedusing hard target labels. Recently, Tang et al. [11] utilized
a similar approach for training RNN with a trained DNN.
However, they had to train two different neural networks
and built different structures to transfer the knowledge frombigger models.
From the probabilistic perspective, Graves et al. [40]proposed a variational inference method for learning themean and variance of Gaussian distribution for each weightparameter. They reformulated the variational inference as
the optimization of a Minimum Description Length [41].
By modeling each weight parameter, they learned the importance of each weight in regard to the model. After the
training process was finished, they pruned the parameters byremoving the weight that has a high probability to be zero.
However, they still needed large matrix multiplication andrepresented their model in dense weight matrix, and thus thealgorithmic and memory complexity remained the same asin the original model. LeCun et al. [42] proposed a method
to prune the weight with low-saliency based on their second
derivatives.
Another approach to tackle the compression problem
by a technical perspective is to limit the precision for weightparameters. Gupta et al. [43] and Courbariaux et al. [44]minimized the performance loss while using fewer bits (e.g.,
16 bits) to represent floating points. Courbariaux et al. [45]proposed BinaryConnect to constrain the weight possible

TJANDRA et al.: RECURRENT NEURAL NETWORK COMPRESSION BASED ON LOW-RANK TENSOR REPRESENTATION

values to -1 or +1. Han et al. [46] utilized a combination
between pruning, quantization and compression by Huffman
coding. Most of these ideas can be easily applied with
our proposed model since several deep-learning frameworks
have built-in low-precision floating point options [47, 48].
Model compression using low-rank matrix has also
been reported [12, 49]. Both of these works showed thatmany weight parameters are significantly redundant, and byrepresenting them as low-rank matrices, they reduced thenumber of parameters with only a small drop in accuracy.
Recently, Lu et al. [50] used low-rank matrix ideas to reduce
the number of parameters in an RNN. Novikov et al. [13] utilized TT-format to represent weight matrices on feedforward
neural networks. From their empirical evaluation on DNNbased architecture, the feedforward layer represented by the
TT-format has a far better compression ratio and smalleraccuracy loss compared to the low-rank matrix approach.
Tjandra et al. [15] and Yang et al. [51] utilized the TT-format
to represent the RNN weight matrices. Based on the empirical results, TT-format are able to reduce the number ofparameters significantly and retain the model performanceat the same time. Recent work from [52] used block decompositions to represent the RNN weight matrices.
To the best of our knowledge, there are only a few research about compression on RNN models. In this work, we
presented an RNN model by using CP decomposition, Tucker
decomposition and TT-decomposition to re-parameterize theweight matrices into a low-rank tensor format. We also compared the performance to standard uncompressed RNNs witha greater number of parameters. We expect our model could
minimize the number of parameters and preserved the performance simultaneously.
8.

Conclusion

In this paper, we presented an efficient and compact RNN
model based on tensor decomposition method. In this work,
we presented some alternatives for compressing RNN parameters with tensor decomposition methods. Specifically,
we utilized CP-decomposition and Tucker decomposition torepresent the weight matrices. For the experiment, first,
we run our experiment on polyphonic music dataset with uncompressed GRU model and three tensor-based RNN models(CP-GRU, Tucker-GRU and TT-GRU). We compare the performance of between all tensor-based RNNs under various
number of parameters. Based on our experiment results, weconclude that TT-GRU has better performances compared toother methods under the same number of parameters. To extend our result, we run another experiment which is speech
recognition task. We modified DeepSpeech2 architecture
and we were able to represent dense weight matrices inside
the RNN layer with multiple low-rank tensors based on TTformat. We evaluated our proposed model on LibriSpeechdata. Our proposed TT-GRU is able to compress the number of parameters significantly while retaining high model
performance and accuracy at the same time.

13

Acknowledgment
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101 and JP17K00237. We also thankTakuma Mori for his supports and insightful discussions.

References[1] J.L. Elman, “Finding structure in time,” Cognitive science, vol.14,
no.2, pp.179–211, 1990.
[2] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol.9, no.8, pp.1735–1780, 1997.
[3] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen,
R. Prenger, S. Satheesh, S. Sengupta, A. Coates, et al., “Deepspeech: Scaling up end-to-end speech recognition,” arXiv preprint
arXiv:1412.5567, 2014.
[4] D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, et al., “Deepspeech 2: End-to-end speech recognition in English and Mandarin,”
arXiv preprint arXiv:1512.02595, 2015.
[5] Y. Wu, M. Schuster, Z. Chen, Q.V. Le, M. Norouzi, W. Macherey,
M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al., “Google’s neuralmachine translation system: Bridging the gap between human andmachine translation,” arXiv preprint arXiv:1609.08144, 2016.
[6] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint
arXiv:1409.0473, 2014.
[7] I. Sutskever, O. Vinyals, and Q.V. Le, “Sequence to sequence learningwith neural networks,” Advances in neural information processing
systems, pp.3104–3112, 2014.
[8] M. Schuster, “Speech recognition for mobile devices at Google,”
Pacific Rim International Conference on Artificial Intelligence, pp.8–
10, Springer, 2010.
[9] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in aneural network,” arXiv preprint arXiv:1503.02531, 2015.
[10] J. Ba and R. Caruana, “Do deep nets really need to be deep?,”
Advances in neural information processing systems, pp.2654–2662,
2014.
[11] Z. Tang, D. Wang, and Z. Zhang, “Recurrent neural network training
with dark knowledge transfer,” 2016 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP), pp.5900–
5904, IEEE, 2016.
[12] M. Denil, B. Shakibi, L. Dinh, N. de Freitas, et al., “Predicting
parameters in deep learning,” Advances in Neural Information Processing Systems, pp.2148–2156, 2013.
[13] A. Novikov, D. Podoprikhin, A. Osokin, and D.P. Vetrov, “Tensorizing neural networks,” Advances in Neural Information Processing
Systems, pp.442–450, 2015.
[14] I.V. Oseledets, “Tensor-train decomposition,” SIAM Journal on Scientific Computing, vol.33, no.5, pp.2295–2317, 2011.
[15] A. Tjandra, S. Sakti, and S. Nakamura, “Compressing recurrent
neural network with tensor train,” Neural Networks (IJCNN), 2017
International Joint Conference on, pp.4451–4458, IEEE, 2017.
[16] A. Tjandra, S. Sakti, and S. Nakamura, “Tensor decomposition
for compressing recurrent neural network,” 2018 International Joint
Conference on Neural Networks (IJCNN), pp.1–8, July 2018.
[17] T. Mori, A. Tjandra, S. Sakti, and S. Nakamura, “Compressing endto-end ASR networks by tensor-train decomposition,” Interspeech
2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 September 2018.,

14
pp.806–810, 2018.
[18] A. Graves, A.r. Mohamed, and G. Hinton, “Speech recognition
with deep recurrent neural networks,” Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on,
pp.6645–6649, IEEE, 2013.
[19] Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” Neural Networks, IEEE
Transactions on, vol.5, no.2, pp.157–166, 1994.
[20] S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber, “Gradient
flow in recurrent nets: the difficulty of learning long-term dependencies,” 2001.
[21] Q.V. Le, N. Jaitly, and G.E. Hinton, “A simple way to initialize recurrent networks of rectified linear units,” arXiv preprint
arXiv:1504.00941, 2015.
[22] J. Martens and I. Sutskever, “Learning recurrent neural networks with
Hessian-free optimization,” Proceedings of the 28th International
Conference on Machine Learning (ICML-11), pp.1033–1040, 2011.
[23] A. Graves, N. Jaitly, and A.r. Mohamed, “Hybrid speech recognition with deep bidirectional LSTM,” Automatic Speech Recognition
and Understanding (ASRU), 2013 IEEE Workshop on, pp.273–278,
IEEE, 2013.
[24] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares,
H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” arXiv
preprint arXiv:1406.1078, 2014.
[25] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluationof gated recurrent neural networks on sequence modeling,” arXiv
preprint arXiv:1412.3555, 2014.
[26] R.A. Harshman, “Foundations of the parafac procedure: Models andconditions for an" explanatory" multimodal factor analysis,” 1970.
[27] H.A. Kiers, “Towards a standardized notation and terminology inmultiway analysis,” Journal of chemometrics, vol.14, no.3, pp.105–
122, 2000.
[28] T.G. Kolda and B.W. Bader, “Tensor decompositions and applications,” SIAM review, vol.51, no.3, pp.455–500, 2009.
[29] L.R. Tucker, “Some mathematical notes on three-mode factor analysis,” Psychometrika, vol.31, no.3, pp.279–311, 1966.
[30] R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training
recurrent neural networks,” International Conference on Machine
Learning, pp.1310–1318, 2013.
[31] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS’10). Society for Artificial Intelligence and Statistics, 2010.
[32] N. Boulanger-lewandowski, Y. Bengio, and P. Vincent, “Modelingtemporal dependencies in high-dimensional sequences: Application
to polyphonic music generation and transcription,” Proceedings ofthe 29th International Conference on Machine Learning (ICML-12),
ed. J. Langford and J. Pineau, New York, NY, USA, pp.1159–1166,
ACM, 2012.
[33] A.L. Maas, A.Y. Hannun, and A.Y. Ng, “Rectifier nonlinearities
improve neural network acoustic models,” Proceedings of the 30th
International Conference on Machine Learning (ICML-13).
[34] M. Bay, A.F. Ehmann, and J.S. Downie, “Evaluation of multiple-f0
estimation and tracking systems.,” 2009 International Society for Music Information Retrieval Conference (ISMIR), pp.315–320, 2009.
[35] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[36] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an
asr corpus based on public domain audio books,” Acoustics, Speechand Signal Processing (ICASSP), 2015 IEEE International Conference on, pp.5206–5210, IEEE, 2015.
[37] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel,
M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky,
G. Stemmer, and K. Vesely, “The kaldi speech recognition toolkit,”
IEEE 2011 Workshop on Automatic Speech Recognition and Understanding, IEEE Signal Processing Society, Dec. 2011. IEEE Catalog

IEICE TRANS. INF. & SYST., VOL.Exx–??, NO.xx XXXX 200x

No.: CFP11SRW-USB.
[38] D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg,
C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, et al., “Deepspeech 2: End-to-end speech recognition in english and mandarin,”
International Conference on Machine Learning, pp.173–182, 2016.
[39] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence datawith recurrent neural networks,” Proceedings of the 23rd international conference on Machine learning, pp.369–376, ACM, 2006.
[40] A. Graves, “Practical variational inference for neural networks,” Advances in Neural Information Processing Systems, pp.2348–2356,
2011.
[41] G.E. Hinton and D. Van Camp, “Keeping the neural networks simple
by minimizing the description length of the weights,” Proceedingsof the sixth annual conference on Computational learning theory,
pp.5–13, ACM, 1993.
[42] Y. LeCun, J.S. Denker, and S.A. Solla, “Optimal brain damage,”
Advances in neural information processing systems, pp.598–605,
1990.
[43] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deeplearning with limited numerical precision,” Proceedings of the 32nd
International Conference on Machine Learning, ICML 2015, Lille,
France, 6-11 July 2015, pp.1737–1746, 2015.
[44] M. Courbariaux, J.P. David, and Y. Bengio, “Training deep neural networks with low precision multiplications,” arXiv preprint
arXiv:1412.7024, 2014.
[45] M. Courbariaux, Y. Bengio, and J.P. David, “BinaryConnect: Training deep neural networks with binary weights during propagations,”
Advances in Neural Information Processing Systems, pp.3123–3131,
2015.
[46] S. Han, H. Mao, and W.J. Dally, “Deep compression: Compressing
deep neural networks with pruning, trained quantization and huffman
coding,” arXiv preprint arXiv:1510.00149, 2015.
[47] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro,
G.S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser,
M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray,
C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals,
P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,”
2015. Software available from tensorflow.org.
[48] Theano Development Team, “Theano: A Python framework forfast computation of mathematical expressions,” arXiv e-prints,
vol.abs/1605.02688, May 2016.
[49] T.N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, “Low-rank matrix factorization for deep neural network
training with high-dimensional output targets,” 2013 IEEE International Conference on Acoustics, Speech and Signal Processing,
pp.6655–6659, IEEE, 2013.
[50] Z. Lu, V. Sindhwani, and T.N. Sainath, “Learning compact recurrent
neural networks,” 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.5960–5964, IEEE,
2016.
[51] Y. Yang, D. Krompass, and V. Tresp, “Tensor-train recurrent neural networks for video classification,” International Conference on
Machine Learning, pp.3891–3900, 2017.
[52] J. Ye, L. Wang, G. Li, D. Chen, S. Zhe, X. Chu, and Z. Xu, “Learningcompact recurrent neural networks with block-term tensor decomposition,” Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pp.9378–9387, 2018.

TJANDRA et al.: RECURRENT NEURAL NETWORK COMPRESSION BASED ON LOW-RANK TENSOR REPRESENTATION

Andros Tjandra
received his B.E degreein Computer Science (cum laude) from Facultyof Computer Science, Universitas Indonesia, Indonesia in 2014. Later, he continued his study
and obtained M.S (cum laude) in 2015 from thesame faculty and university. At the moment,
he is currently taking doctoral course in Graduate School of Information Science, Nara Institute of Technology, Japan. He is a student
member of ASJ. His research interests include
machine learning (deep learning), speech recognition, speech synthesis and natural language processing.

Sakriani Sakti
received her B.E. degree
in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In
2000, she received DAAD-Siemens Program
Asia 21st Century Award to study in Communication Technology, University of Ulm, Germany,and received her MSc degree in 2002. During
her thesis work, she worked with Speech Understanding Department, DaimlerChrysler Research Center, Ulm, Germany. Between 20032009, she worked as a researcher at ATR SLC
Labs, Japan, and during 2006-2011, she worked as an expert researcher
at NICT SLC Groups, Japan. While working with ATR-NICT, Japan, she
continued her study (2005-2008) with Dialog Systems Group University ofUlm, Germany, and received her PhD degree in 2008. She was actively
involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003-2007), A-STAR and U-STAR (2006-2011). In 2009-2011,
she served as a visiting professor of Computer Science Department, University of Indonesia (UI), Indonesia. From 2011, she has been an assistantprofessor at the Augmented Human Communication Laboratory, NAIST,Japan. She served also as a visiting scientific researcher of INRIA ParisRocquencourt, France, in 2015-2016, under "JSPS Strategic Young Researcher Overseas Visits Program for Accelerating Brain Circulation". She
is a member of JNS, SFN, ASJ, ISCA, IEICE and IEEE. Her research interests include statistical pattern recognition, speech recognition, spoken
language translation, cognitive communication, and graphical modelingframework.

Satoshi Nakamura
is Director of Data
Science Center and Professor at the Graduate
School of Science and Technology, Nara Institute of Science and Technology, Japan, Team
Leader of Riken AIP Tourism Information Analytics Team, and Honorarprofessor of Karlsruhe
Institute of Technology, Germany. He received
his B.S. from Kyoto Institute of Technology in1981 and Ph.D. from Kyoto University in 1992.
He was Associate Professor of Graduate Schoolof Information Science at Nara Institute of Science and Technology in 1994-2000. He was Director of ATR Spoken
Language Communication Research Laboratories in 2000-2008 and Vice
president of ATR in 2007-2008. He was Director General of Keihanna
Research Laboratories and the Executive Director of Knowledge Creating Communication Research Center, National Institute of Information andCommunications Technology, Japan in 2009-2010. He is currently Director of Augmented Human Communication laboratory and a full professor
Nara Institute of Science and Technology, Japan. He is interested in modeling and systems of speech-to-speech translation and speech recognition.
He is one of the leaders of speech-to-speech translation research and has
been serving for various speech-to-speech translation research projects in15

the world including C-STAR, IWSLT and A-STAR. He received Yamashita
Research Award, Kiyasu Award from the Information Processing Societyof Japan, Telecom System Award, AAMT Nagao Award, Docomo Mobile
Science Award in 2007, ASJ Award for Distinguished Achievements in
Acoustics. He received the Commendation for Science and Technology by
the Minister of Education, Science and Technology, and the Commendationfor Science and Technology by the Minister of Internal Affairs and Communications. He also received LREC Antonio Zampolli Award 2012. He
has been Elected Board Member of International Speech Communication
Association, ISCA, since June 2011, IEEE Signal Processing Magazine Editorial Board Member since April 2012, IEEE SPS Speech and Language
Technical Committee Member since 2013. He is ATR Fellow, IPSJ Fellow
and IEEE Fellow.