Title: RIEMANNIAN STOCHASTIC GRADIENT DESCENT FOR TENSOR-TRAIN RECURRENT NEURAL NETWORKS

Abstract: The Tensor-Train factorization (TTF) is an efficient way to compress large weight matrices of fully-connected layers and recurrent layers in recurrent neural networks (RNNs). However, high Tensor-Train ranks for all the core tensors of parameters need to be element-wise fixed, which results in an unnecessary redundancy of model parameters. This work applies Riemannian stochastic gradient descent (RSGD) to train core tensors of parameters in the Riemannian Manifold before finding vectors of lower Tensor-Train ranks for parameters. The paper first presents the RSGD algorithm with a convergence analysis and then tests it on more advanced Tensor-Train RNNs such as bi-directional GRU/LSTM and Encoder-Decoder RNNs with a Tensor-Train attention model. The experiments on digit recognition and machine translation tasks suggest the effectiveness of the RSGD algorithm for Tensor-Train RNNs.

1. INTRODUCTION:

Recurrent Neural Networks (RNNs) are typically composed of large weight matrices of fullyconnected and recurrent layers, thus massive training data as well as exhaustive computational resources are required. The Tensor-Train factorization (TTF) aims to reduce the redundancy of RNN parameters by reshaping large weight matrices into high-dimensional tensors before factorizing them in a Tensor-Train format Oseledets (2011) . The notation of Tensor-Train usually suggests that TTF is applied for the tensor representation of model parameters. Tensor-Train was initially applied to fully-connected layers Novikov et al. (2015) , and it has been recently generalized to recurrent layers in RNNs such as LSTM and GRU Yu et al. (2017) . Compared with other tensor decomposition techniques like the CANDECOMP/PARAFAC decomposition Kolda & Bader (2009) and Tucker decomposition Kim & Choi (2007) , Tensor-Train can be easily scaled to arbitrarily high dimensions and have the advantage of computational tractability to significantly large weight matrices.

Given a vector of Tensor-Train ranks r = (r 1 , r 2 , • • •, r d+1 ), TTF decomposes a d-dimensional

tensor W ∈ R (m1•n1)×(m2•n2)×•••×(m d •n d )

into a multiplication of core tensors according to (1), where the k-th core tensor C [k] ∈ R r k ×m k ×n k ×r k+1 , and any index pair (i k , j k ) satisfies 1 ≤ i k ≤ m k , 1 ≤ j k ≤ n k . Additionally, the ranks r 1 and r d+1 are fixed to 1. W((i 1 , j 1 ), (i 2 , j 2 ), ..., (i d , j d )) = C [1] (r 1 , i 1 , j 1 , r 2 )C [2] (r 2 , i 2 , j 2 , r 3 )•••C [d] (r d , i d , j d , r d+1 ) (1) Thus, when the TTF technique is applied to the fully-connected (FC) layer with feed-forward weight matrix W , a tensor W is firstly converted from W and is then decomposed to a multiplication of the core tensors as shown in (2), where C [t] is the t-th core tensor, X denotes a tensor of input, B refers to a tensor of bias, the tensor of outputs Ŷ ∈ R n1×n2×•••×n d , and σ is a sigmoid function. For clarity, the notation T T L(W , X) is used to simplify the representation of a Tensor-Train fully-connected layer, which is shown in (3).

Ŷ(j 1 , j 2 , ..., j d ) = σ( m1 i1=1 • • • m d i d =1 d t=1 C [t] (r t , i t , j t , r t+1 )X(i 1 , i 2 , ..., i d ) + B(j 1 , j 2 , ..., j d )) (2) Ŷ = σ(T T L(W , X)).

(3) Likely, we use (4) to represent an RNN with a feed-forward weight matrix W and a recurrent weight matrix U . In (4), X t is an input matrix at time t, h t-1 and h t separately denote the hidden vectors of time t -1 and t, and σ refers to the sigmoid function.

EQUATION

The largest benefit from Tensor-Train models is the capability to reduce the model parameters tremendously. For example, a Tensor-Train FC layer needs i m i • n i • r i • r i+1 parameters in total. In comparison, the total number of parameters of an FC layer is about

O( i m i • i n i ),

which is much larger than the associated Tensor-Train FC one.

The Tensor-Train models are found widespread. For example, Yang et al. (2017) Contributions. We summarize the key contributions of this paper as follows:

• This work applies RSGD to iteratively find vectors of lower Tensor-Train ranks with the update of parameters in the training process. The RSGD algorithm and the related theoretical analysis are also presented. • We design Bi-directional Tensor-Train GRU/LSTM Chung et al. (2014) , and Encoder-Decoder Tensor-Train RNNs with a Tensor-Train Attention mechanism Luong et al. (2015) .

We apply the RSGD algorithm to the Tensor-Train RNN models on the digit recognition and machine translation tasks.

To the best of our knowledge, this is the first work that applies RSGD to train Tensor-Train RNNs to find the optimal core tensors of parameters with vectors of lower Tensor-Train ranks. Moreover, this is the first work that builds Tensor-Train RNNs with complex architectures for natural language processing (NLP) tasks.

2. RIEMANNIAN STOCHASTIC GRADIENT DESCENT:

The optimization problem of Tensor-Train RNNs can be formulated as a Riemannian optimization problem as shown in (5), where {X, Y } is a data sequence with length T , W represents a tensor of parameters which lies in a d-dimensional Riemannian Manifold (M, µ) with a Riemannian measure µ , and the Tensor-Train ranks for core tensors of W must be element-wise no higher than the vector r = (r 1 , r 2 , ..., r d+1 ) as shown in (5).

min W∈M f (W; X, Y ) s.t., tt rank(W) ≤ r.

(5)

Besides, the Riemannian measure µ induces an inner product structure in each tangent space T x M associated with a tensor x ∈ M . Specifically, ∀u, v ∈ T x M , the inner product < u, v >= µ x (u, v).

Algorithm 1 Riemannian Stochastic Gradient Descent 1. Given the labeled input data (X, Y ) with sequence length T , and the learning rate η.

2. W (0) ← the randomly initialized core tensors {C [1] , C [2] , ..., C [d] }.

3. For t = 1, 2, ..., T : 4.

For i = 1, 2, ..., d: 5.

Choose a gradient g

C [i] = ∇ C [i] f (W (t) ; X, Y ) in the tangent space T C [i] M . 6. Âi = C [i] -ηg C [i] . 7. Ĉ[i] ← Exp C [i] ( Âi ). 8. ( Ĉ[i] , ri ) ← rounding( Ĉ[i]

).

9.

W (t) ← { Ĉ[1] , Ĉ[2] , • • • Ĉ[d] }. 10. r ← (r 1 , r2 , ...r d ). 11.

Reshape the core tensors {C [1] , C [2] , ..., C [d] } based on the updated r. 12. Return W (T +1) = {C [1] , C [2] , • • •, C [d] }.

Similarly, µ induces the norm of u ∈ T x M as ||u|| = µ x (u, u) ≥ 0. In addition, the µ induced inner product and the norm preserve the basic properties like definiteness, homogeneity and triangle inequality.

Algorithm 1 presents the RSGD Algorithm. The algorithm mainly consists of two main procedures: one is the update of parameters in the tangent space and conducting an exponential mapping, and the second one is the rounding to lower Tensor-Train ranks. As illustrated in Figure 1 , step 5 firstly obtains a gradient g C [i] on a tangent space T Ci M at the core tensor C [i] in Riemannian Manifold (M, µ), and step 6 conducts a gradient descent on the tangent space to generate a new tensor Âi .

Step Figure 2 : The rounding procedure.

The exponential mapping in Algorithm 1 is formulated in (6). Unfortunately, it is not easy to solve the problem because we have to deal with the calculus of variations, or we have to know the Christoffel symbols Lubich et al. (2015) . Therefore, a fast and straightforward retraction method is applied as a first-order approximation to the exponential mapping as shown in Algorithm 2. min

EQUATION

Algorithm 2 The Retraction Algorithm 1. Given the core tensors {C [1] , C [2] , • • •, C [d] } and the Tensor-

Train rank r = {r 1 , r 2 , • • •, r d+1 }. 2. For i = 1 to d: 3. A i ← reshape C [i] by the shape of (r i • n i • m i ) × r i+1 . 4. For i = 1 to d: 5.

Âi , Σ i ← QR decomposition(A i ).

6.:

C [i] ← reshape Âi by (r i , n i , m i , r i+1 ).

7.:

A i+1 ← multiply(Σ i , A i+1 ), if i < d. 8. Return {C [1] , C [2] , • • •, C [d] }.

Algorithm 3 The Rounding Algorithm 1. Given the core tensors {C [1] , C [2] , • • •, C [d] } and a constant maximum rank r max . 2. Initialize r = {1, 1, ..., 1}.

3.. For:

i = 1 to d: 4. A i ← reshape C [i] by the shape of (r i • n i • m i ) × r i+1 . 5. For i = 1 to d: 6. col num(C [i] ) = n i • m i • ri+1 . 7. row num(C [i] ) = r i • r i+1 ri+1 . 8. ri+1 = min(r max , col num(C [i] ), row num(C [i] )). 9. (U i , S i , V i ) = SVD(A i ). 10. Ûi = U i [:, 1 : ri+1 ], Ŝi = S i [1 : ri+1 , 1 : ri+1 ], Vi = V i [1 : ri+1 , :].

11.:

Âi = Ûi Ŝi Vi .

12. C [i] ← reshape Âi as R ri×ni×mi×ri+1 .

13. Return the updated core tensors {C [1] , C [2] , • • •, C [d] }.

Algorithm 2 presents the retraction algorithm. The main idea of the retraction algorithm is to orthogonalize the core tensors {C [1] , C [2] , • • •, C [d] } in a left-to-right order by the QR decomposition.

The rounding algorithm is shown in Algorithm 3. Similar to the retraction algorithm, a left-to-right tensor matricization is firstly initialized. Then, the Tensor-Train rank is updated before conducting an SVD computation. The returned core tensors {C [1] , C [2] , • • •, C [d] } are based on the updated vector of lower Tensor-Train ranks. Furthermore, the rounding procedure has the property presented in Proposition 1.

Proposition 1. The rounding procedure of Algorithm 3 does not change values of the objective function f for the Tensor-Train RNNs. That is, for a tensor x ∈ S r , the rounding tensor x ∈ S r, we have f (x) = f (x).

Proof. Given the weight matrix W with core tensors {C [1] , C [2] , • • •, C [d] }, for an input tensor X ∈ M , we obtain (7) and (8) according to (1).

EQUATION

which suggests that a vector of Tensor-Train ranks determines a submanifold for generating core tensors of the tensor W, but the values of the objective functions are invariant to the change of the vector of Tensor-Train ranks, obtaining Proposition 1.

3. CONVERGENCE ANALYSIS:

This section analyzes the convergence of the RSGD algorithm. The necessary definitions and theorems are firstly introduced, and the analysis is then provided. Since the objective functions of Tensor-Train RNNs are always geodesically non-convex Zhang et al. (2016) , we only consider the convergence of RSGD for non-convex cases. Definition 2. A different function f : M → R is geodesically L-smooth if its first-order gradient is geodesically L-Lipchitz continuous. Specifically, ∀x, y ∈ M we have 9.

EQUATION

)

where g x is the sub-gradient of f (x) at x in the tangent space T x M , and Exp -1 x (y) is the inverse exponential mapping which projects the curve line in M d from x to y back to the gradient x in the tangent space T x M .

From the RSGD algorithm (Algorithm 1), it is not hard to find the sub-gradient g x = ∇f (x) and Exp -1

x (y) = -η∇ x f (x), and thus Theorem 3 can be derived. Theorem 3. For a differentiable and geodesically L-smooth function f , the Riemannian stochastic gradient descent algorithm ensures (10).

EQUATION

)

where T refers the total iterations, x 0 and x * denote the initial and the optimal points respectively, η is the learning rate, and ||Exp -1 x (y)|| 2 is bounded by η 2 k 2 .

Proof. Assume x * and x 0 separately refer to the optimal and initial points. For all x t and x t+1 at two consecutive times, we can derive (11) based on Definition 2.

EQUATION

By applying Exp -1 xt (x t+1 ) = -η∇f (x t )) and E[||Exp -1 xt (x t+1 )|| 2 ] ≤ η 2 k 2 , we obtain (12), where xt+1 is the rounding tensor of xt+1 . The Proposition 1 ensures that f (x t+1 ) = f ( xt+1 ).

EQUATION

)

By summing the two sides of equation 12 from 0 to T -1, we derive (13).

EQUATION

EQUATION

)

After rounding x * to x * , we finally obtain the result 15 of Theorem 3.

EQUATION

)

Furthermore, Theorem 3 suggests that the number of iterations T satisfies ( 16) before reaching the convergence. 17) to ( 24) present the functional mechanisms of the Bi-directional Tensor-Train GRU, where the pairs ( -→ r t , ←r t ) and ( -→ z t , ←z t ) denote forward and backward update gate operations, respectively, and the pair (

EQUATION

- → d t , ← - d t )

represents the forward-backward reset gate operations. In addition, the pair (

-→ h t , ← - h t )

is the memory cell holding information of last times. o h is the output of the Bi-directional Tensor-Train GRU which combines -→ h t and ←h t by concatenating them before feeding through a Tensor-Train linear layer.

EQUATION

EQUATION

In addition, we also design a Bi-directional Tensor-Train LSTM which involves more operational gates than the Bi-directional Tensor-Train GRU.

4.2. THE TENSOR-TRAIN ENCODER-DECODER RNNS WITH ATTENTION MODELS:

The Encoder-Decoder RNNs are commonly used in sequence-to-sequence deep learning applications. Moreover, the attention mechanism significantly improves the performance of Encoder-Decoder RNNs Vaswani et al. (2017) .

The Bi-directional Tensor-Train RNN like GRU or LSTM is used to construct the Encoder-Decoder architecture. Moreover, we set up the Tensor-Train Attention model in addition to the Tensor-Train Encoder-Decoder RNNs. Thus, the entire model is built on Tensor-Train layers.

To build a Tensor-Train Attention model, it is necessary to add the Tensor-Train layer to generate an Attention vector as shown in ( 25), where c t is a context vector (26) with attention weights α ts (27), a t is the output of the Attention model at time t, hs denotes a vector built from the outputs of the forward and backward stages, and h t refers to the output of the hidden layers at time t. 

EQUATION

5. APPLICATIONS:

This section first introduces the implementation of the Tensor-Tensor RNNs. Then, we present two applications where the RSGD algorithms were tested. One application is the digit recognition task on the sequential MNIST dataset; the other is the task of machine translation on the Multi30K dataset Elliott et al. (2016) .

5.1. IMPLEMENTATIONS:

We employed PyTorch to implement our Tensor-Train RNNs. The data structures of our implementations were partly built on the free Tensor-Train toolkit implemented by the tool Tensorflow Novikov et al. (2018) . However, we employed the tool PyTorch to take the advantage of dynamic graph generation, which is much more useful for NLP tasks.

5.2. DIGIT RECOGNITION ON THE MNIST DATASET:

The first application is the digit recognition task on the MNIST dataset. The dataset consists of 60000 data with 28 * 28 pixels for each digital image. Instead of vectorizing the image pixels into a long vector as an input for a static deep neural network, image pixels are taken as data sequence where the time step is set to 28, and the input dimension is set to 28. In our experiments, the training and testing sets were separately composed of 50000 and 10000 data. 2000 data were selected from the training set for building a validation set, and they were not included in the training set.

As for the experimental setup, we applied both Bi-directional Tensor-Train GRU and Bi-directional Tensor-Train LSTM to the task. The dimension of the Manifold (M, µ) was set to d = 3, and the vector of Tensor-Train ranks was initialized with high and shared values r = (1, 10, 10, 1) for the Tensor-Train layers. The weight matrix of the input-hidden Tensor-Train layer was converted to the tensor with the shape (2 × 7 × 2) by (6 × 6 × 6), and the weight matrix of the hidden-hidden Tensor-Train layer was converted to the tensor with the shape (6 × 6 × 6) by (6 × 6 × 6). The RSGD algorithm with a learning rate 0.01 was applied to both Bi-directional Tensor-Train GRU/LSTM. The experimental results are shown in Figure 4 , and the statistics of parameters of hidden layers are shown in Table 1 . The results in Figure 4 suggest that the Tensor-Train Encoder-Decoder RNN performs closer or even better than the Encoder-Decoder one, although the convergence speed of the Tensor-Train model is relatively slower in the first several iterations. On the other hand, Table 1 shows that RSGD leads to a further decrease in the number of parameters. 

:



Figures and Tables:

7 projects Âi back to Ĉ[i] in Riemannian Manifold (M, µ) by an exponential mapping. Finally, as shown in Figure 2, the rounding function in step 8 transforms Ĉ[i] in the submanifold S r to the core tensor Ĉ[i] with a vector of lower Tensor-Train ranks r in a new submanifold S r. Note that the vectors of Tensor-Train ranks r and r span two submanifolds S r ⊂ M and S r ⊂ M respectively.After that, the next iteration of the the parameter update is conducted in S r.

Figure 1: An exponential mapping.Figure2: The rounding procedure.

) 4 ADVANCED TENSOR-TRAIN RNNS This section introduces the Tensor-Train RNNs with the advanced architectures used in this work. The new Tensor-Train architecture is based on the Bi-directional Tensor-Train GSU/LSTM. The other one is the Tensor-Train Encoder-Decoder RNNs with the Tensor-Train Attention mechanism. 4.1 BI-DIRECTIONAL TENSOR-TRAIN GRU/LSTM Firstly, we introduce the Bi-directional Tensor-Train GRU/LSTM, which involves twice more parameters than the Tensor-Train GRU/LSTM. Equations from (

The results are shown in Figure3, where we compared the Bi-directional Tensor-Train GRU/LSTM with the traditional Bi-directional GRU/LSTM regarding recognition error rates and number of parameters. Inspecting the recognition error rate, the Bi-directional Tensor-Train GRU obtains a result that is close to that of the traditional Bi-directional GRU/LSTM, and the performance of the Bidirectional Tensor-Train LSTM becomes a bit worse. Regarding comparison with the number of parameters, both Bi-directional Tensor-Train GRU/LSTM can significantly reduce the number of parameters by taking only 1% parameters of the Bi-directional GRU/LSTM at final. Notably, the RSGD algorithm further reduces the number of parameters of Bi-directional GRU/LSTM by lowering the Tensor-Train ranks.

Figure 3: Experimental results on the sequential MNIST dataset ('TT' denotes Tensor-Train, 'params.' means the number of parameters, 'init.' means the initial, and 'final' refers to the final number).

The Bi-directional Tensor-Train GRU was used to build the Encoder-Decoder architecture, and a Tensor-Train Attention model was added to the architecture. The RSGD algorithm with a learning rate 0.01 was applied to the Tensor-Train Encoder-Decoder RNNs with the Tensor-Train Attention model. An initial vector of Tensor-Train ranks was set as r = (1, 6, 6, 6, 1), and the weight matrix of hidden layers was converted to the tensor with the shape of (4 × 4 × 4 × 4) by (4 × 4 × 4 × 4). Besides, the baseline was based on the traditional Bi-directional GRU and LSTM-based Encoder-Decoder architecture with the Attention model, where the Stochastic Gradient Descent algorithm with learning rate 0.001 was used to update parameters.

Figure 4: Experimental results of Machine Translation on the Multi30K dataset (PPL refers to the logarithm of loss values).

set up Tensor-Train Recurrent Neural Networks for video classification, andYu et al. (2017) applied the Tensor-Train as an End-to-End dynamic model for multi-variate forecasting environmental data. However, those applications focused on simple deep learning architectures, and a vector of Tensor-Train ranks r is element-wise fixed. When more complex deep models with numerous Tensor-Train layers are involved, it is not easy to find appropriate Tensor-Train ranks for each core tensor. However, when applying a vector of shared and high Tensor-Train ranks to all parameters of Tensor-Train layers, each of them has a vector of lower Tensor-Train ranks, which results in an additional decrease in the number of parameters.

Table content: <table><tr><td>This work applies Riemannian Stochastic Gradient Descent (RSGD) to the RNN Tensor-Train lay-</td></tr><tr><td>ers. Unlike Stochastic Gradient Descent (SGD), which conducts the update of parameters in the</td></tr><tr><td>Euclidean space, RSGD finds optimal core tensors of parameters associated with a vector of lower</td></tr><tr><td>Tensor-Train ranks in Riemannian Manifold, thereby decreasing the number of total parameters of</td></tr><tr><td>Tensor-Train RNNs.</td></tr><tr><td>Our work is inspired by Bonnabel (2013) and Lubich et al. (2015), which are two excellent in-</td></tr><tr><td>troductions to the theory of Stochastic Gradient Descent on Riemannian Manifolds. Additionally,</td></tr><tr><td>Zhang et al. (2016) was a recent work that proposed a fast stochastic optimization on Riemannian</td></tr><tr><td>Manifolds. Besides, Absil et al. (2009) is a useful reference of Riemannian optimization.</td></tr></table>

Statistics of Model ParametersThe next application is a machine translation task from Dutch to English on the Multi30K dataset. In the dataset, there are separately 29000, 1014, and 1000 sentence pairs for training data, validation data, and test data, respectively.

Table content: <table><tr><td>Models</td><td>Parameters of hidden layers</td></tr><tr><td>Bi-directional Encoder-Decoder</td><td>462144</td></tr><tr><td colspan="2">Tensor-Train Bi-directional Encoder-Decoder (initial) 15154</td></tr><tr><td>Tensor-Train Bi-directional Encoder-Decoder (final)</td><td>11088</td></tr><tr><td colspan="2">5.3 MACHINE TRANSLATION ON THE MULTI30K DATASET</td></tr></table>

Peer review (iclr2019):
 This paper proposes an algorithm for optimizing neural networks parametrized by Tensor Train (TT) decomposition based on the Riemannian optimization and rank adaptation, and designs a bidirectional TT LSTM architecture. I like the topic chosen by the authors, using TT to parametrize layers of neural networks proved to be beneficial and it would be very nice to exploit the Riemannian manifold structure to speed up the optimization. But, the paper needs to be improved in several aspects before being useful to the community. In particular, I found the several mathematical errors regarding basic definitions and algorithms (see below the list of problems) and I’m not happy with lack of baselines in the experimental comparison (again, see below). The math problems 1) In equations (1), (2), (7), and (8) there is an error: one should sum out the rank dimensions instead of fixing them to the numbers r_i. At the moment, the left-hand side of the equations doesn’t depend on r and the right-hand side does. 2) In two places the manifold of d-dimensional low-rank tensors is called d-dimensional manifold which is not correct. The tensors are d-dimensional, but the dimensionality of the manifold is on the order of magnitude of the number of elements in the cores (slightly smaller actually). 3) The set of tensors with rank less than or equal to a fixed rank (or a vector of ranks) doesn’t form a Riemannian (or smooth for that matter) manifold. The set of tensors of rank equal to a fixed rank something does. 4) The function f() minimized in (5) is not defined (it should be!), but if it doesn’t have any rank regularizer, then there is no reason for the solution of (5) to have rank smaller then r (and thus I don’t get how the automatic rank reduction can be done). 5) When presenting a new retraction algorithm, it would be nice to prove that it is indeed a retraction. In this case, Algorithm 2 is almost certainly not a retraction, I don’t even see how can it reduce the ranks (it has step 6 that is supposed to do it, but what does it mean to reshape a tensor from one shape to a shape with fewer elements?). 6) I don’t get step 11 of Alg 1, but it seems that it also requires reshaping a tensor (core) to a shape with fewer elements. 7) The rounding algorithm (Alg 3) is not correct, it has to include orthogonalization (see Oseledets 2011, Alg 2). 8) Also, I don’t get what is r_max in the final optimization algorithm (is it set by hand?) and how the presented rounding algorithm can reduce the rank to be lower than r_max (because if it cannot, one would get the usual behavior of setting a single value of rank_max and no rank adaptivity). 9) Finally, I don’t get the proposition 1 nor it’s proof: how can it be that rounding to a fixed r_max won’t change the value of the objective function? What if I set r_max = 1? We should be explained in much greater detail. 10) I didn’t get this line: “From the RSGD algorithm (Algorithm 1), it is not hard to find the sub-gradient gx = ∇f(x) and Exp−1 x (y) = −η∇xf(x), and thus Theorem 3 can be derived.” What do you mean that it is not hard to find the subgradient (and what does it equal to?) and why is the inverse of the exponential map is negative gradient? 11) In general, it would be beneficial to explain how do you compute the projected gradient, especially in the advanced case. And what is the complexity of this projection? 12) How do you combine optimizing over several TT objects (like in the advanced RNN case) and plain tensors (biases)? Do you apply Riemannian updates independently to every TT objects and SGD updates to the non-TT objects? Something else? 13) What is E in Theorem 3? Expected value w.r.t. something? Since I don’t understand the statement, I was not able to check the proof. The experimental problems: 1) There is no baselines, only the vanilla RNN optimized with SGD and TT RNN optimized with your methods. There should be optimization baseline, i.e. optimizing the same TT model with other techniques like Adam, and compression baselines, showing that the proposed bidirectional TT LSTM is better than some other compact architectures. Also, the non-tensor model should be optimized with something better than plain SGD (e.g. Adam). 2) The convergence plots are shown only in iteration (not in wall clock time) and it’s not-obvious how much overhead the Riemannian machinery impose. 3) In general, one can decompose your contributions into two things: an optimization algorithm and the bidirectional TT LSTM. The optimization algorithm in turn consist in two parts: Riemannian optimization and rank adaptation. There should be ablation studies showing how much of the benefits come from using Riemannian optimization, and how much from using the rank adaptation after each iteration. And finally some typos / minor concerns: 1) The sentence describing the other tensor decomposition is a bit misleading, for example CANDECOMP can also be scaled to arbitrary high dimensions (but as a downside, it doesn’t allow for Riemannian optimization and can be harder to work with numerically). 2) It’s very hard to read the Riemannian section of the paper without good knowledge of the subject, for example concepts of tangent space, retraction, and exponential mapping are not introduced. 3) In Def 2 “different function” should probably be “differentiable function”. 4) How is W_c represented in eq (25), as TT or not? It doesn’t follow the notation of the rest of the paper. How is a_t used? 5) What is “score” in eq (27)? 6) Do you include bias parameters into the total number of parameters in figures? 7) The notation for tensors and matrices are confusingly similar (bold capital letters of slightly different font). 8) There is no Related Work section, and it would be nice to discuss the differences between this work and some relevant ones, e.g. how is the proposed advanced TT RNN different from the TT LSTMs proposed in Yang et al. 2017 (is it only the bidirectional part that is different?) and how is the Riemannian optimization part different from Novikov et al. 2017 (Exponential machines), and what are the pros and cons of your optimization method compared to the method proposed in Imaizumi et al. 2017 (On Tensor Train Rank Minimization: Statistical Efficiency and Scalable Algorithm). Please, do take this as a constructive criticism, I would be happy to see you resubmitting the paper after fixing the raised concerns!