Language Modeling Using Tensor Trains
Authors: Zhan Su
Yuqin Zhou
Benyou Wang
Qiuchi Li
Jakob Grue Simonsen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Tensor network
RNNs
Language modeling

Abstract: Tensor networks have previously been shown to have potential in language modelling in theory, but lack of practical evidence support.  We propose a novel Tensor Train Language Model (TTLM) based on Tensor-Train decomposition.  We prove that TTLM generalizes  Second-order Recurrent Neural Networks (RNNs),  Recurrent Arithmetic Circuits and Multiplicative Integration RNNs in the sense that the architecture of all of these are, essentially, special cases of that of TTLM. To show the usefulness of TTLM, we perform a principled experimental evaluation on language modeling tasks, showing that our proposed variants, TTLM-large and TTLM-Tiny, can be more effective than Vanilla RNN while TTLM-Tiny has the half of the model size.

Under review as a conference paper at ICLR 2023
LANGUAGE MODELING USING TENSOR TRAINS
Anonymous authors
Paper under double-blind review
ABSTRACT
Tensor networks have previously been shown have potential in language model- ing in theory but lack practical evidence support. We propose novel Tensor Train Language Model (TTLM) based Tensor-Traee decomposition. To show the use- fulness of TTLM, we perform principled experimental evaluation on real-world language modcling tasks, showing that our proposed variants. TTLM-Large and TTLM-Tiny, can be more effective than Vanilla RNNs with low-scale of hidden sizes. Also, we demonstrate the relationship between TTLM and Second-order Recurrent Neural Networks (RNNs), Recurrent Arithmetic Circuits, and Multi- plicative Integration RNNs in the sense that the architectures of all of these are, essentially, special cases of that of TTLM
INTRODUCTION
A language model assigns probabilities of sequences words from the vocabulary V: the number of texts increases exponentially w.r.t to length N. Hence the domain of language model is, by def- inition, the exponential space V. However, due to the vanilla exponential space being intractable, existing work tends use recurrent or auto-regressive architectures generate conditional prob- abilities based on the context (typically encapsulated as a fixed-length dense vector). This indeed simplifies the calculation.
Recently, researchers (Pestun & Vlassopoulos, 2017: Miller et al. 2021 Zhang et al. 2019) have
reconsidered the view of language models joint probabilities of text, it leads to exponential representations in tensor space. Word connections could be preserved in the exponential tensor space when measuring joint probabilities. To deal with the exponential space complexity, mathematical tool called 'tensor network' has been used to reduce the exponential space of language modeling tractable one (Pestun & Vlassopoulos 2017). However, so-called 'tensor network language model' in Pestun & Vlassopoulos (2017) is currently a concept that needs to be proved practically.
As proof-of-conceyy work, we derive a Tensor Train Language Model (TTLM) (the simplest tensor network). Technically, we represent sentence based on the exponential semantic space constructed by the tensor product of word representations. The probability the sentence is obtained by the inner product of two high- nigh-dimensional tensors: the input &e (X) and the global coefficients A.
Under the framework of TTLM, we propose two variants: TTLM-TTiy and TTLM-Large. Also, clarify the relationship between the proposed TTLM and series of Recurrent Neural Networks (RNNs) (i.e., Second-order RNNs (Goudreau et al., 1994), Recurrent Arithmetic Circuits (RACs) (Levine al. 2018), and Multiplicative Integration RNNs (MI-RNNs) (Wu et al. (016)). These connections open a new eye to understanding RNNs and give some natural implementations for TTLM.
We benchmark these TTLM variants and analyze the difference in their working mechanism and behaviors. Experimental results on language modeling tasks show that our TTLM variants could outperform than Vanilla-RNNg under the same training setting. These demonstrate the feasibility of TTLM.
The main contributions of our work can be summarized as follows:
1 'The code is available at wnthtts://// / gi thub. wwm/ www./t/eooottaiinnniiiooo....eae
n1m
TTensor networks are, roughly, decompositions of large tensors into sets of smaller tensors and have been employed in physics, mathematics, and machine learning (Sun et al.]|l..000 [020 Novikov et al. 2015) Cohen et al. 2016. Stoudenmire Schwab 2016b Cheng et al. 2019] Novikov al. 2016 Selvan & Dam 2020

---

Under review as a conference paper at ICLR 2023
1. We propose novel Tensor Train Language Model, first attempt to apply tensor net- works on real-world language modeling tasks.
2. We propose two novel TTLM variants, TTLM-Large and TTLM-Tiny, and theoretically demonstrate the relationship between TTLM and series of existing RNNs.
3. Compared Vanilla-RNNs on WikiText-2 and PTB datasets, TTLM-Large reduces per- plexity by 14.3 and 16.0, respectively, and TTLM-Tiny reduces perplexity by 1.7 and 8.5, respectively.
RELATED WORK
Previous studies tensor networks in machine learning have mainly been devoted to analyzing the theoretical properties of neural networks. A better understanding of feed-forward, convolutional and recurrent architectures has been gained, including compression parameters (Novikov et al., 2015),
expressive power (Cohen et al. 2016; Cohen & Shashua 2016 Khrulkov et al. 20088.. and depth
efficiency for long-term memory (eevine al. 2018)).
Focusing on natural language modeling, certain studies have tensorized existing network architec- tures (Novikov et al. 2013). while few studies have applied tensor networks alone as a language model. To the best of our knowledge, tensor network language models have remained a theoretical proposal instead of an empirical model (Pestun Vlassopoulos, [007 Pestun al., (017). Perhaps, Miller et al. (2021) 1) is the only other work that uses tensor networks for probabilistic sequence mod- eling, while fails to scale up its model for real-world sequence modeling tasks. We first derive tensor network language model in the way that it can be applied to real-world language modeling datasets, while its variants outperform Vanilla RNNs with lower-scale hidden sizes.
PRELIMINARIES
We briefly recapitulate basic notions and notations Fl full technical introductions can be found in standard textbooks (e.g.. Bi et al. (2022): Itskov (2009)).
Notation. For the purposes of this paper, every lensor A is multidimensional array of elements (called components) of IR, each denoted by its integer coordinates in the array: e.g., for two- dimensional array, the component at position j.j N is denoted A.j. The order of tensor is how many indices it has (c.g.. vector is a first-order tensor, a matrix M is second-order tensor, etc.). The dimension of tensor refers to the number of values that particular index (or so-called mode) can take, e.g., the dimension of B E R/x22x is Jn X In x Iz-
Tensor product. For two tensors E R41 (order j) and D 11.511, (order k), their tensor product is denoted by @ and return a tensor Eigre Cig..dy Distrriiss (order j + k).
Generalized inner product. For two tensor X, Y R4 11xx1xx--xxx of the same size, their inner product is defined as (X,Y) Y) -Ei E' xxxxxxx"xxxxxxxxxx and Y E RR1x/xx-iixx Txxt, sharing N modes of the same size, the "general-
IN Eiz=t Xi..22.... Yt, Yt..k.....kx For two tensors
ized inner product" defined in Koossafi et al. 2020) is calculated as
Y) = Σ
da YIgdg
m1
with (X, N)N RI.,
4 LANGUAGE MODELING USING TENSOR TRAINS
We introduce language model in tensor space in Sec. 4.1 We define our general Tensor Train
Language Model in Sec. 4.2 and its special case, TTLM, in Sec. 4.3
Most of the notations here follow the textbook Deep Learning Goodfellow et al. 2016
2

---

Under review as a conference paper at ICLR 2023
3)
Vector
Matrix
Order-3 tensor
Order-N tensur
b) Tensor Contraction
[4]
B
1 [4]]
12 El [Aa]
B
Ix-1 [xx=a] [x E [fx]
E [h]
EE [6]
E [22
E [4]
Gy
Byu
Figure I: A quick introduction to tensor diagram notation. There are two rules of tensor diagrams. (1) tensors are notated by solid shapes with a number of 'legs' corresponding to their indices. (2) connecting two index lines implies contraction or summation over the connected indices. In this paper, we augment our equations with these diagrams make them easier to visualize.
4.1 LANGUAGE MODELS IN A TENSOR SPACE
Natural language typically has complex dependencies between features (e.g., tokens or words) (Hou 2013] that are not captured well by standard methods such as feature concatenation. One could also see similar interaction between any arbitrary features in factorization machines (Rendle)
2010). Suppose given text consists of N words X = [z(1), .2(2), [N))] and let fi RR be feature extractor (it can be one-hot encoding or word embedding). We now define representation of X designed to capture these dependencies:
(X)) = fx @ fa(x ... (N(
f(((1))
where the tensor space is T & Rd 3 ... & RIN Each component f; represents independent meaning-bearing units, such morphemes or latent factors. For simplicity, we assume that a text shares the same one-hot encoding f (t) E I VI in later sections. Consequently, (X) is a |V|*- dimensional tensor that records all possible combinations words X.
Inspired by Kossaifi et al. (2020g; Zhang et al. (2019), we define a tensor regression model to obtain the estimated probability for each text X:
p(x) (A, ((X))
M
Σ
Anera +(X).....
where (-) denotes the inner product of two same-sized tensors, and A is a regression weight tensor of the same shape as ((X) in the tensor space VON
V8 ... X V where V refers to RRV] Similar
functions were considered in Novikov et al. 2016); Stoudenmire & Schwabl 2016a 2018); Zhang et al. (2019).
Khrulkov al.
4.2 GENERAL TENSOR TRAIN LANGUAGE MODEI.
Suppose the sequence of indices of words in the text X is W1, W2, WN. where w; E 21,2,-- [VB and its corresponding weight in A is denoted Aw, ww We use TT decompo- sition to represent Awr in the TT format (Oseledets) 2011] as follows:
Such dependencies (including collocation) have been viewed analogy of entanglement Hou et al.
2013

---

Under review as a conference paper at ICLR 2023
a) ITLM
b) b)TTLM-Ting Tiny
cJ LTLM-Large
R
W
G
M
M
[V]
(X)
(2
[P]
Figure 2: a) Tensor Train Language Model based on Eq. 5 b) TT core TTLM-TTi c) TT core of Lare-Large. The dashed line in the square represents A, + (X), or G. Note that the only difference between TTLM-Large and TTLM-Tiny Tiny is whether tensor W
Aw W2...
(1)
(2)
(N)
1xR1 Rg X
RN-1X1
(3)
RV-1
Σ
Σ
Σ
(
( Ganezas
(N) No
22
ON 1=1
where the tensors (1) E BR cores.
(t 1,... d. Ro = By = 1 by definition) are called TT
We now combine (X) and A in the TT format to define general TTLM. The elements Sans
in
Eq. 3 can be represented
V Σ f(x
G. Ga
(4)
where each f(((1) is a one-hot vector having we for at most one t, and has zeros elsewhere. Therefore, one-hot encoding enables us to integrate the input data into the TT format of A by inserting Eq. into Eq, 3
IVI
R
Rx-1
A WN
Σ
Σ 01=1
Σ
)(xl1)
f((())
f(o(N))
G
$1. d
ON-1==
Ri
R2
Rx
=
Guian
Ga aawaa
2)
(N) ()
GaN-
(5)
a1=1 02=2
Gx-1=1
(A, (A, (())
where ((X) = x f()()). The difference between Eq. and Eq. that Eq. has combined A
and ((X) in the low-dimensional form. This is because that Eq. 3 can compute the elements of A
Oseledets 2011). and because ((X) here is the tensor product of one-hol vectors, so that Eq. 5 can
compute Eq. 2 Further, since Eq. [5 now has input data (one-hot vectors) and weights (TT cores),
we name Eq. 5 as our general 77LM.
4.3 TTLM
Here we consider a special class general TTLM. Despite its site-dependent TT cores G (*) tially giving it more expressiveness for language modeling, this property currently generates unnec- essary obstacles to its applicability, like the choice of R, We here provide detailed explanation of its special case: TTL.M.
poten-

---

Under review as a conference paper at ICLR 2023
Definition.
Suppose all the intermediate
TT cores equal
to each other G E RI in Eq. Then, TTLM is defned
G
G
(N-1) E XVXX and G
G (N)
as follows:
M
p(X)=
Σ
Σ
f( )i Gha f(x(2)) Goriza Ma((N)
G
GaN-ieN
(N)
frem on -11
Σ
Gwia
Gay
Gan May--wwN-ax 11 GaN-IeN
G
where its tensor diagram notation is shown in Figure 2h.
Recursive information. We recursively unfold the calculation of TTLM Eq. 6 and find that G has two sources of "input": the information from the previous recursive unfolding, and the input data f (xx(1) (see Eq. 15 for a detailed version). From this perspective, G acts as a bilinear map G R V x R + R and we can regard the information in the previous step as a hidden state (t)
M given by:
h
(t) TTLM
f(z"0)"" (t-1)
(7)
where (x)()). G, and h fILM are contracted together (we permute the indices of G from x to R V
RxR which does not change the number of indices).
Recursive Probability Computation. Here, we here provide further details about the process of computing (XX by TTLM in practice.
In language modeling, p(X X) often decomposed using
the chain rule (Bahl et al. (983) as follows:
CC
69
N
pXX)
= II p(x ((x(0)((((:t11))) t=1
IP]
[V]
(202)
(2
where (1it-1) denotes the text (x((), _r22......)) .... (.(-1)).
At time t, the output prediction a model, (() V. is a probability distribution word (²)) given (1:t-11,
Figure Recursive calculation of con- ditional probability in TTLM. Here provide an example that given the text (1:3), ((
In TTLM, we define (t) as follows:
softmax ( h (8) where ()()) V is the probability distri- bution of word (4))
y (t) softmax (G("NTN ())
where (f) E R X is the last TT core in TT format time t. Such definition the same that of RNNs, which use hidden states and a weight matrix to calculate word probabilities. Fig. provvidss simple example.
We derive the definition of (t) in high-dimensional space, if we substitute h (1-1) h in Eq.
by
Eq. 6 and Eq. 7
V
y softmax
Σ
Σ
{(x(1)); (1())))))) Gapiges ... Ga
(t)
(9)
1=1
softmax
(A(1:t)),
(()(0))
(10)
softmax (A:1:)) T (1:(111111))
(11)
where (I:t) E you (1x+1=))) vot-1 and (:)-1 denotes the "generalized inner product
defined" in Preliminary. Note that Eq. 9 is the low-dimensional form of Eq. [[] similarly to the
relationship between Eq. and Eq.
5

---

Under review as a conference paper at ICLR 2023
By these definitions, there some interesting properties of TTLM. 1) We teacher forcing Juraffsky 2000) learn parameters of TT cores. The hidden-to-output tensor G ()) defined to
be the same as the input-to- tensor G @ 3) and G have no parameters in common. We provide detailed explanation of the relationship between different TT cores in Appendix A
TTLM VARIANTS
To show the versatility and practical applicability of the TTLM framework, propose two new variants: TTLM-Large and TTLM-Tiny in Sec. 5.I In Sec. 5.2 we briefly summarize the relationship between TTLM and some widely-used RNNs.
5.1 NEW VARIANTS: LTLGG-L AND TTLM-TINY
The TT core G in TTLM is an entire third-order tensor. In the two variants, we decompose G into several separate tensors without violating the TT format, as shown Fig. 2b and Fig. D We define TTLM-Tiny and Laree as follows:
h Tiny
= (() (010)/TT> " 5w hi h (t-1)
(12)
h Lange
(0) ( wn sw (1-2)
where W E [RIV is the input-to-hidden tensor: ; W E RR Rxx n. and E Rxxxxxx fourth-order diagonal tensor such that &ijia = 1 iff the i =j=
= k= and &iykt otherwise.
The relationship between our proposed models and TTLM as follows: W in both models take the same role as (*) in TTLM (i.e. input-to-hidden and hidden-to-output), while G w wW in TTLM-Tiny and G we sh in TTLM-Large.
As in RNNs, we compute conditional probability recursively for TTLM-Large and TTLM-Tiny as:
softmax (ePtmax(VVPhhC) (Vh(())
(13)
where V E RRRX/VXXR is output embedding tensor, and P RRRRRXR is projector tensor. Then we tie the input tensor W to the output embedding tensor (we provide detailed explanation in Appendix C
One obvious advantage of our models is to utilize information from the hidden layer and input data separately. Such interaction, particularly TTLM-Tiny, can potentially avoid overfitting, similarly to Wu et al. (2016) where multiplication integration between two sources of "input" can outperform many other methods. In Sec 6.2
we provide relevant experimental evidence.
5.2 EXISTING TTLLM VARIANTS
Given the fact that TT scores of TTLM can vary, Appendix B provides detailed illustration that three existing models, namely second-order RNNs, RACs and MI-RNNs can be considered as one of the "special" implementations of TTLM.
We briefly summarize the differences between the three models: 1) Second-order RNNs use the third-order T as the TT cores with activation function given Eq. 14 2) RACs use Whz @ Wh
as the TT cores given Eq. 18 3) MI-RNNs use Whe @ Whh as the TT cores with an activation function given Eq. 19
Along with our two proposed models, we study the experimental performance of second-order RNNs, RACs and MI-RNNs compared to TTLM-Large and TTLM-Tiny in Section 6
EXPERIMENTAL EVALUATION
To further understand the properties of TTLM variants, investigate the effectiveness of TTLM-Large and TTLM-Tmy compared to Second-order RNNs, RACs, MI-RNNs, and Vanilla- RNNs. We conduct experiments from two distinct perspectives: (1) The rank of TT decomposition

---

Under review as a conference paper at ICLR 2023
Wiki
PTB
Hidden
Layer
Embed Size
Model
Params
PPL.
Params
PPI.
(Rank)
RNN Miikolov & Zweie 2012
124.7
300
1
LSTM
Zaremba et 2014)
114.5
200
2
L.TTM
Grave
20T6)
99.3
82.3
1024
1
LSTM
(Merty et al.
2017
100.9
80.6
2
Vanilla-RNNs
11.6M
96.6
4.0M
115.3
20
1
400
Second-order RNNs*
11.8M
96.0
4.2M
108.2
20
400
RACs*
11.6M
97.6
4.0M
116.8
400
MI-RNNs
11.6M
99.6
4.0M
119.1
400
TILM
12.2M
546.4
4.2M
559.8
20
400
TTLM-Time Tiny
11.6M
94.9
4.0M
106.8
20
1
400
TTLM-Large
11.8M
82.3
4.2M
99.3
400
Table 1: PPL evaluation on set on WikiText-2 and PTB. Models tagged with indicate that they are re-implemented by ourselves. The symbol means these data are not available in their original paper. Params are the training parameters, the details are in Appendix
has been proved to be the dimension of the hidden states of RNNs (Khrulkov et al. 2018). Here, we study the influence of rank on the effectiveness of our TTLM variants in Sec 6.2 (2) In Sec 6.3 we analyse the influence of non-linearity for TTLM variants.
6.1 EXPERIMENTAL SETTING
Tasks, Datasets, and Metrics. We conduct experiments on word-level language model datasets: English Penn Treebank (PTB), which consists of 929k training words, 73k validation words, and 82k test words. It has 10k words in its vocabulary (Marcinkiewicz) 1994). The WikiText-2 dataset (Merity al. 2016) derived from Wikipedia articles and consists 2088k training words, 217k validation words, 45k test words, and vocabulary of over 30,000 words. We compare these models on the language modelling task, evaluated by the Perplexity (PPL) (Meister & Cotterell 2021): the lower the perplexity, the better the model.
Baselines. Vanilla RNNs, Second-RNNS, RACs and MI-RNNs are our baselines. We also provide some original results RNN-based models as references (Mikolov & Zweig [012]] Zaremba al.) 2014 Grave et al. 2016 Merity al. 2017). The implementation details are introduced in Ap-
pendix C
compare the effectiveness comparable models in the same scale: 1) We set the rank/hidden size of TTLM AariantsVVanillaRRNN as [5, 10, 20, 25, 30, 35, 40, 45, 50]. The embedding size of these models is the squared number of hidden sizes/ranks. 3) To avoid the impact of the large embedding size the model performance, also provide several common choices of emhedding size Vanilla-RNNs by setting its embedding size as [100, 200, 300] (we name them as RNNs-100, RNNs-200, RNNs-300 correspondingly and use them in Fig. 55_ 4) The random seed is fixed to ensure the experimental results are not influenced by initializing the weights. 5) We train all models for 50 epochs and choose the best model in the validation set.
Hyperparameters.
predict the result in the test
6.2 RANK AND EFFECTIVENESS ANALYSIS
The rank of the TT format has been used to explain the expressive power long-term memory capacity RNNs (Khrulkov et al.) 2018 Levine al.) 2018). However, the relationship between rank and effectiveness in language modelling has yet to be shown practically. Later, will evaluate the effectiveness of our models w.r.t rank.
Effectiveness. (1) Fig. shows the influence of the rank on our models based on the validation PPL. The validation PPL of TTLM-Large drops down at the early training step but easily increases when the rank increases. In contrast, the validation PPL of TTLM-Tiny stably decreases as the rank increases. (2) Compared to Vanilla-RNNs, the influence of the rank
models based the test

---

Under review as a conference paper at ICLR 2023
300
rank-10
rank-10
rank-20
Validation PPL
rank=30
Validation PPL
rank=30
250
rank=40
rank=40
200
200
150
150
20
Epoch
Epoch
Validation perplexity TTLM-Large
(b) Validation perplexity TTLM-
Figure 4: Rank analysis for the Laree and TTLM-Tiny on PTB.
PPL is shown Fig. 5 Our models outperform Vanilla-RNNa on all used parameter settings. (3) Table provides supplementary example to show the comparison of models with baselines and some references. TTL.M-Large and M-TNy are more effective than baselines.
Overfitting. Based on Fig. 4a and Fig. 4p. we find that TTLM-Large is more prone to overfitting than TTLM-Tiny. As the only difference between the two models is W this suggests that the simpler parame- terization of the TT cores, the more easily the model avoids overfitting. This finding is consistent with the comparison between MI-RNNs and Second-order RNNs by Wu et al. (2016).
180
RNNs
160
RNNs-200
RNNs-300
Test PPL
Large
140
TTLM-Tiny
120
100
10
30
Low-scale. Despite the effectiveness of our models under the current hyperparameter settings, Fig. 5 re- veals their limited potential when the rank is larger than 40 where the test perplexity of Vanilla-RNN bly decreases. Therefore, we expect that our model can outperform vanilla RNNs with low-scale of hidden size (i.c. range from 5 to 50), but not larger scales; this clear tradeoff of using the simple parameterization of TT cores as we do in TTLMs.
Hidden Size (Rank)
Figure 5: Comparison of test perplex- ity on PTB. RNNs here Vanilla-RNNs. and its embedding size is the same as TTLM-Large and TTLM-Timy, RNNs- 100, RNNs-200 and RNNs-300 the RNNs with fixed embedding sizes of 100, 200 and 300, respectively.
still sta-
6.3 NON-LINEARITY ANALYSIS.
Fig. shoows the effects of the tanh activation function on TT variants based on validation PPL. Re- garding the speed of convergence, tanh speeds up 1TL.M-Large-tanl ITLLM-Tinyttan,e MI-RNNs while barely influencing second-order RNNs. Regarding the magnitude of the lowest validation perplexity, tanh impairs the performance TTLM-Large and TTLM-Tiny, but has little influence on multiplicative integration and the third-tensor T in Second-order RNNs.
Thus, the influence of lineaiiy on TTLM variants depends on TT cores settings, both for the convergence of validation PPL and the magnitude of the lowest validation PPL. Thus, from an experimental point of view, the effect non-linearity functions on one TT variant cannot simply be transferred or analogized to another TT variant. This also suggests that one should be wary of the analogy between tensor decomposition and existing neural network models at the implementation level declared by previous research (Khrulkov et al. 2018 Levine al. 2018). The activation function could factor to influence such an analogy.
7 CONCLUSION
We first apply TT decomposition to real-world language modeling and name the framework as TTLM. We propose variants: TTL.M-Lare and TTT. TTLM-Tiny, and show that they ef-
8

---

Under review as a conference paper at ICLR 2023
TTLM-Large TTLM-Large-tanh
TTLM-Tiny
180
180
TTLM-Tiny TTLM-Tiny-tanh
Validation PPL
Validation PPL
160
160
120
120
100
20
40
100
training step
training step
(a) TTLM-Large v.s. TTLM-Large-tan
(b) TTLM-Tiny v.s. TTLM- TTLM-Tiny-tamm
350
Second-linear
300
MI-RNNs
Second-order RNNs
Validation PPL
300
250
Validation
200
200
150
150
20
40
60
100
0
20
40
60
80
100
training step
training step
(c) RACs v.s. MI-RNNs
(d) Second-lineas v.s. Second-orde order RNNs
Figure 6: Comparison of influence of non-linearity on TTLM variants on PTB The suffix -tanh refers to model using the tanh activation function. Second-linear refers to Second-order RNNs without activation function.
fective compared Vanilla-RNNG with low-scale of hidden sizes. Meanwhile, we demonstrate that Second-order RNNs, RACs and MI-RNNs are special implementations of TTLM.
REFERENCES
Lalit R Bahl, Frederick Jelinek, and Robert L Mercer. A maximum likelihood approach to contin- uous speech recognition. IEEE transactions on pattern analysis and machine intelligence, (2): 179-190, 1983.
Yingyue Bi, Yingcong Lu, Zhen Long, Ce Zhu, and Yipeng Liu. Chapter tensor decomposi-
tions: computations, applications, and challenges. In Yipeng Liu (ed.), Tensors for Data Process-
ing, pp. 1-30. Academic Press, 2022. ISBN 978-0-12-8844447-0 doi: https://doi..rg//0..u 10166
B978-0- 1777-00-12-824444-000000-5 URL httppppwwwww.....eeeeeeeee...ooa wtw.://iiieet.ccoocccccceeeeeeee
atticle/pi1/B970012222
Song Cheng, Lei Wang, Tao Xiang, and Pan Zhang. Tree tensor networks for generative modeling. Physical Review B, 99(15): 2019.
Nadav Cohen and Amnon Shashua. Convolutional rectifier networks generaiized tensor decompo- sitions. 2016. doi: 10.4850/ARRXX..16030001 /Rt..ttass./aavv..ot////aa////// 00162
Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: tensor analysis. In Conference on learning theory, pp. 698-728. PMLR, 2016.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Mark W Goudreau, Lee Giles, Srimat T Chakradhar, and Dong Chen. First-order versus second- order single-layer recurrent neural networks. IEEE Transactions Neural Networks, 5(3):511- 513, 1994.
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with continuous cache. arXiv preprint 16Xiv:16/2.044420 2016.

---

Under review as a conference paper at ICLR 2023
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Yuexian Hou, Xiaozhao Zhao, Dawei Song, and Wenjie Li. Mining pure high-order word asso- ciations via information geometry for information retrieval. ACM Transactions on Information Systems (TOIS), 31)::1133, 2013.
Mikhail Itskov. Tensor Algebra and Tensor Analysis for Engineers: With Applications to Continuum Mechanics. Springer Publishing Company, Incorporated, 2nd edition, 2009. ISBN 3540939067.
Dan Jurafsky. Speech & language processing. Pearson Education India, 2000.
Valentin Khrulkov, Alexander Novikov, and Ivan Oseledets. Expressive power recurrent neural
networks. In International Conference Learning Representations, 2018. URL https:// openreview wtt.:e/ooooommddd...............
Jean Kossaifi, Zachary C Lipton, Arinbjöm Kolbeinsson, Aran Khanna, Tommaso Furlanello, and Anima Anandkumar. Tensor regression networks. The Journal of Machine Learning Research, 21(1):4862 2020.
Yoav Levine, Or Sharir, and Amnon Shashua. Benefits of depth for long-term memory of recurrent networks. 2018.
Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. Using Large Corpora, 273, 1994.
Diego Maupomé and Marie-Jean Meurs. Language modeling with general second-order RNN. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4749-4753, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546- 34-4. URL https:: //c www.aaannoooooyy..ooooooooooo wwwpllog..rgg//00ooo/. rec 1oocc-1.55 584
Clara Meister and Ryan Cotterell. Language model evaluation beyond perplexity. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 5328-5339, Online, August 2021. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1eXX:v:600.000 07843, 2016.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm lan- guage models. arXiv preprint arXiv: 708.000000
1708. 028800212, 2017.
Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pp. 239-239. IEEE, 2012.
Jacob Miller, Guillaume Rabusseau, and John Terilla. Tensor networks for probabilistic sequence modeling. In International Conference on Artificial Intelligence and Statistics, pp. 3079-3087. PMLR, 2021.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Alexander Novikov, Mikhail Trofimov, and Ivan Oseledets. Exponential machines. arXiv preprint arXiv:1605.03795, 2016.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295- 2317, 2011.
Vasily Pestun and Yiannis Vlassopoulos. Tensor network language model. arXiv:1710.10245 2017.
arXiv preprint
Vasily Pestun, John Terilla, and Yiannis Vlassopoulos. Language as a matrix product state. arXiv preprint arXiv:171 171..01416, 2017.
10

---

Under review as a conference paper at ICLR 2023
Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.000 05859, 2016.
Guillaume Rabusseau, Tianyu Li, and Doina Precup. Connecting weighted automata and recurrent neural networks through spectral learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1630-1639. PMLR, 2019.
Steffen Rendle.
Factorization machines. In 20/0 IEEE International conference on data mining, PP- 995-1000. IEEE, 2010.
Raghavendra Selvan and Erik B Dam. Tensor networks for medical image classification, In Medical Imaging with Deep Learning, pp. 721-732. PMLR, 2020.
Edwin Stoudenmire and David J Schwab.
Supervised learning with tensor net- works. In D. D. Lee. M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 29, pp. 4799-4807 Curran Associates, Inc., 2016a. URL http://papers: nips. ww/ppppp........
Edwin Stoudenmire and David Schwab. Supervised learning with tensor networks. Advances in Neural Information Processing Systems, 29, 2016b.
Zheng-ZIi Zhi Sun, Cheng Peng, Ding Liu, Shi-Ju Ran, and Gang Su. Generative tensor network classification model for supervised machine learning. Physical Review B, 101(7):075135, 2020.
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural net-
works. In ICML, 2011.
Paul Werbos. Backpropagation through time: what does and how to do it. Proceedings of the IEEE, 18000), 1990.
Ronald Williams and Jing Peng. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural computation, 2(4):490-501, 1990.
Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan Salakhutdinov. On Multi- plicative Integration with Recurrent Neural Networks, November 2016. URL http::/arax
org/ (otg/abs16606 06630 (rXiv:1606.06630 [cs].
Wojeiech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv: 40..4099.2999 2014.
Lipeng Zhang, Peng Zhang, Xindian Ma, Shuqin Gu, Zhan Su, and Dawei Song. A generalized language model in tensor space. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7450-7458, 2019.

---

Under review as a conference paper at ICLR 2023
APPENDIX
A RELATIONSHIP BETWEEN TT CORES TTLM
To help readers understand the roles of TT cores in TTLM, we here provide detailed calculation
of the probability of a text X [x(1) (2) [x0]] by TTLM. Note that all the intermediate TT cores are equal to each other:
G (2) G GNN-D) and G (
The calculation of ( (i.e. the conditional probability of xx(1) given x((:-11) at time t) can be described as three steps. As step I, suppose f(x(1) is one-hot vector having f(x(1) 1. The
calculation of G ()) f(x(1) in TTLM as follows:
f (100)
G
G
11
R
G (('((())))))
(x(1))
G
(1)
2
(2()) M
1212
[ G
PTTLM
(f-1) As step II, TTLM will calculate fx:(1) Gh Gh TTLM where E (2, ,t t- 1}. For example, h
is calculated in Eq. 7 time = 2 as follows:
TTM (2) GG
As step III, TTLM will output ())) as follows:
G
(0)
(t-1)
GL
TTLM
(t-1)
en
KTTLM,
(t)
M11
v22
V R
(t-1)
(t-1) TTLMR
Observing the calculation, G1) G and (()) theoretically have no parameters in common (though we set (!) = (*) for simplicity). Further, their roles in TTLM are different: (!)) can be viewed as word embedding matrix: G deals with two sources of information, i.e. hidden state and input word; G (t) extracts the evidence provided in (t-1) and generates a set of scores over vocabulary.
B RELATIONSHIP BETWEEN TTLM AND SOME RNNS
We now demonstrate the relationship between TTLM and Second-ordes RNNs, Recurrent Arith- metic Circuits (RACs) and Multiplicative Integration RNNs (MI-RNNs).
To avoid symbol clutter when representing different RNNs, the notation is: Who E Rx denotes the input-to-hine hidden matrix, hh e lttlt denotes hidden-to-hidden matrix, Φ(-) is a element-wise nonlinear activation function. Also, different hidden states are denoted as: Second-order RNNs (1) (1) b RACS (h ( and MI-RNNs (1.())
(h
12

---

Under review as a conference paper at ICLR 2023
B.1 RELATION TO SECOND-ORDER RNNS
Unlike Vanilla-RNNs (Mikolov & Zweig, 2012) that have additive blocks, Second-order RNNs have interaction between hidden states and input data in multiplicative form. This is achieved by a third-order tensor T with the i-th coordinate the hidden states ha defined as (Hochreiter & Schmidhuber 1997 Maupomé & Meurs 2020):
2nd, = (f(x"))) Te., (t-1) + b)
(14)
where T...: E MxR is the ith slice tensor T M and is bias vector. For simplicity, we will ignore for other variants of RNNs since b can be seen as Oth component of f(x(1) which equals to 1. Rabusseau al. (2019) has provided that Tensor Trains generalize linear Second- order RNNs. We here provide basic proof from the perspective of recursive property in TTLM.
Claim B.1. The third-order tensor T Second-order RNNs equals the TT cores in TTLM. The hidden states of Second-order RNNs is identical to that of TTLM if they are accompanied by an activation function.
Proof. The proof is based on the following observation: We recursively unfold the calculation of TTLM in Eq. 5
M
Auj WN Σ [
),,
VI
Σ
Σ f(o
)is f Go
(15)
I
f(x G.o, f(x Garizaz ... f(x G(N)
G
Observe in the above. that at each time step. G has two sources of "input": the information from the previous recursive unfolding (e.g., in the second line, the first line is the previous information), and the input data (x)) From this perspective, acts as a bilinear map RV x R. - R and can regard the information in the previous line hidden state HTTL given by:
IVI
R
Graaca N ((rtt))
PTTMM
(16)
where we permute the indices of Gay-iia, as Giaaaa-1 (note that this does not change the number of indices).
We can also represent the hidden states in Second-order RNNs shown by Eq. 14 in element-wise fashion:
Te...
[V]
(17)
f(x)0))) Tysch (t-1)
=
(t) where j. k the dummy indices it, Ocy specifies the coordinate of (t) just like ar for hTTMM- Thus, and G are the same-sized trainable bi-linear map.
After demonstrating that the third-order tensor T in Second-order RNNs equals the TT cores G. the only difference between the hidden states in Eq. 17 and in Eq. i6 an activation function. If we add an activation function for h TILM the hidden states of Second-order RNNs and TTLM are identical, as shown in Fig. hb.
13

---

Under review as a conference paper at ICLR 2023
Second order RNNs
hy RACS
c) MERNNS
IV
900
Figure 7: a) Second-order RNNs under TTLM framework. b) Hidden state of RACs under TTLM framework. c) hidden state of MI-RNNs under TTLM framework. The dashed line in the square denotes A, (XX) or G. The small hollow circles denote the activation functions.
B.2 RELATION RACS AND MI-RNNS
We here focus Multiplicative Integration (MI), way connect two sources of inputs by the Hadamard product 'O'. MI has been used in RACs, Multiplicative RNNs (M-RNNs) (Sutskever
al. 2011 and MI-RNNS:
Recurrent Arithmetic Circuits (RACs) are recurrent networks with hidden states FRAC defined as (Levine et al. 2018):
h (0 RAC
Wha whrf(((") whh (t-1)
(18)
where these hidden states also used intermediate term in M-RNNs.
Multiplicative Integration RNNs (MI-RNNs) are RACs with an activation function and hidden states (t) Mi defined as (Wu et al. 2016
MI
w( f(x(1)) w (1"1))
(19)
Claim B.2. Given the condition the TT-scores: W Whh, The hidden states of RACs are identical to that of TTLM. The hidden states of MI-RNNs are identical to that of TTLM if they are accompanied by an activation function.
Proof. The proof is based on the following observation: In the language of tensor contractions, Eq. 18 involves contracting the input weights matrix hh with the input vector f fxx()), and contracting the hidden weights matrix Wh with h h The Hadamard product of the two is third-order diagonal tensor 8 e RxRxxx such that dizk iff the = j = and 8,jk = otherwise. Thus, we can write Eq. [8 in element- wise fashion:
VI a
(1) BRACUE
Σ
Σ
[(x
why Sjnc What BRACCE-
f= 1 o==1
(20)
M1
R
Σ
f(x
Gi
a1=1
where = hhz Whh In this case, the hidden state of TTLM in Eq. 16 equal to the hidden
state of RACs in Eq. 20 as shown in Fig, 7p. Similarly, if Eq. 16 is accompanied with activation
function d, Eq. is equal to the hidden state of MI-RNNs Eq 19 as shown in Fig. 7c.
Given Claim B. and B.2 the three models shall be simulated by TTLM with non-lineen activation function and leave finding theoretical proof of this conjecture future work.
14

---

Under review as a conference paper at ICLR 2023
C IMPLEMENTATIONS
We implement all RNNS models, TTLM, LTLgeL and TTLM-Tiny using PyTorch. The weights in the models are adjusted minimize the average cross entropy loss over training se- quences via stochastic gradient descent computed using the truncated backpropagation through time algorithm. (Werbos 1990 Williams & Peng 1990)).
For RNNs, there five matrix parameters: Wiee E &R& is the input embedding matrix, Web REXH is the embedding-to- hidden matrix, W E R is the hidden-to-hiddee matrix. We tie (share the same training parameters) the input embedding Wa and output embedding V which has been proved lead to significant reduction in perplexity (Press & Wolf 2016). So there is projection matrix P [ before the output embedding. All this process introduced in (Press & Wolf 2016).
For TTLM models, we tie the imput tensor T and V. The implementation of is functioned by reshape function, so the interaction between hidden and input can be computed by matrix product. We also let (1)) have the same parameters along the dimension [V] (i.e. G is simplified into GO) RIxR and can be viewed as initial hidden state h ()).
Model
Training Parameters Wee E REXIV Weh IR EXH Wit ERRXH
Vanilla-RNN
PE HXX V E TEREX EXIVI
MI-RNNs
Wh E REXI W R REXH
W ERRXH
PER R V FF
RACs
W0e E R Weh E R EEH Wit ERRXH
PER PERHXE VERR REXVI TE RIIHHI
Second-order RNNs
Wae E REX
TERIXHIII W ERAX
P PEREXE VER EEREII
TTLM
G E R ERAVV G E ER
TTLM-Tiny
TT RRIIVIRR Wht ERAXE, PERRXRXR VER RRXRRII
TTLM-Large
W IR R W BRRRRRRR Wht RRR V
PER V RRXXXIV
Table 2:
Training parameters in our implementation. E' is the embedding size, H/ is the hidden size RNNs, and is the rank in the TTLM. We set Hl = and E R2 to make the parameters of all models in the same scale. The parameters of W and W uniformly initialized in the interval [-0.1, 0.1), Web, W and Wh uniformly initialized between
VT VH
15


Reviews:
[{"review_id": "wyTo_pHN1_i", "reviewer": "Reviewer_meG2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors propose a Tensor Train Language Model (TTLM) to apply tensor networks for language modelling. Also, they prove that TTLM generalizes Second-order RNNs, RACs and MI-RNNs.", "summary_of_the_review": "In this work, the authors propose a Tensor Train Language Model (TTLM) to apply tensor networks for language modelling. Also, they prove that TTLM generalizes Second-order RNNs, RACs and MI-RNNs.\n\nHowever, the novelty of this work is not very high and the reproducibility is uclear.", "strength_and_weaknesses": "Strength\n1. the paper is well written, containing all of the Algebra background of tensor network and tensor train.\n\n2.  they prove that TTLM generalizes Second-order RNNs, RACs and MI-RNNs.\n\nWeakness,\n1. Applying Tensor Train to RNN is not very new. For example,\n\nYang, Yinchong, Denis Krompass, and Volker Tresp. \"Tensor-train recurrent neural networks for video classification.\" International Conference on Machine Learning. PMLR, 2017.\n\nActually, TT is a very common compression technique, and is widely used with CNN, Transformer.\nMa, Xindian, et al. \"A tensorized transformer for language modeling.\" Advances in neural information processing systems 32 (2019).\n\nTherefore, the novelty of this work is not very high. \n\n2. The code is not available, and the reproducibility is uclear due to a lots of details in the proposed framework.\n\n3. It may be better to compare with the popular Transformer to see more benefit of the TTML", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oVwIdY9Y6xe", "reviewer": "Reviewer_CPvM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper gives an experimental characterization and some theoretical connections regarding language models built on recurrent tensor train (TT) models, a form of tensor network (TN). Although TNs have received an increasing amount of interest within machine learning, with TTs having been proposed as promising language models, this appears to be the first work to actually evaluate the performance of such models in language modeling.", "summary_of_the_review": "The experimental contributions of the paper are important, and the theoretical results prove several useful relationships between previous model families. However, the lack of clarity in the description of the model and experiments, along with the absence of significantly novel architectural or theoretical contributions, detract from the paper's score.", "strength_and_weaknesses": "+ Giving concrete experimental results in language modeling is an important step for research into the use of TNs in machine learning. These models have many interesting properties and capabilities that aren't shared by neural nets, but (largely owing to the origins of TNs in physics and mathematics) prior work in this area has skewed heavily towards theory. I want to stress to other reviewers that even though the empirical results reported are a far cry from that of modern neural language models, the fact that simple recurrent TT models are in the same ballpark as pre-Transformer state of the art RNNs is an important result in itself.\n\n+ The authors prove concrete connections between recurrent TT language models and several prior models, namely recurrent arithmetic circuits, second-order RNNs, and multiplicative integration RNNs. These connections aren't surprising and the proofs are basic, but it is nonetheless good to clearly state these connections so that they can be appreciated by the broader machine learning community.\n\n- Many important implementational details are unspecified or unclear, and the paper feels hastily written. I address these points in more detail in the following section, but given that the primary contribution of this work is experimental, ensuring that these experiments are well-explained and fully reproducible should be a priority.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "caAXo_HPtJD", "reviewer": "Reviewer_K2sM", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a model for language modeling. The proposed method models the probability of a certain word sequence as the inner product between a feature vector which is a tensor product of feature vectors for each word, and a learnable weight tensor which is in tensor train (TT) format. The paper claims to show that the proposed method is a generalization of three other RNN-based language models (it's not in two of the three cases), and also runs some numerical experiments.", "summary_of_the_review": "The proposed method is poorly explained. The claimed theoretical results are incorrect. The paper is full of typos and notational inconsistencies. This is a clear reject for me.", "strength_and_weaknesses": "**--- Strengths ---**\n\nNo particular strengths stood out to me when reviewing this paper.\n\n**--- Weaknesses ---**\n\n**W1.** The paper claims (e.g., in the abstract and the conclusion) to show that the proposed TTLM is a generalization of 2nd-order RNNs, RACs, and MI-RNNs. The TTLM model takes the form given in Eq (2). In order to say that TTLM generalizes another model, that model must be expressible in the form given in Eq (2). However, according to Claim 4.1 and 4.2, in order to be able to express 2nd-order RNNs and MI-RNNs additional nonlinearities need to be inserted into the formula in Eq (2), therefore yielding a model which is *not* expressible as a TTLM (i.e., as in Eq (2)). Therefore, it is *incorrect* to say that TTLM generalizes those models! If TTLM truly generalized these models, there shouldn't be a need to further modify Eq (2).\n\nThis is a bit like saying that the set of linear functions (i.e., matrices) generalize functions $f : \\mathbb{R}^{m} \\rightarrow \\mathbb{R}^{n}$ of the form $f(x) = \\sigma(A x)$, where $A$ is a matrix $\\sigma$ is a non-linear functions.\n\n**W2.** The paper is full of typos, missing/redundant commas and periods, and notational inconsistencies that a careful proof reading should have caught; there are two inconsistencies already in the first sentence in Sec 1.\n\n**W3.** On page 1, you say that \"we propose a novel Tensor Train Language Model, as a first attempt to apply tensor networks on language modeling tasks\". This makes it sound like applying tensor networks, and tensor trains in particular, to language modeling is new. But that's not the case. For example, Miller et al. (2021) use matrix product states (which is the same as tensor trains) with language modeling as a potential application. Your contribution needs to be clarified.\n\n**W4.** The paper is difficult to follow in several places. For example:\n- The discussion in Sec 3.3 doesn't make sense. You insert Eq (4) into Eq (3) to get Eq (5)---which is the same as Eq (3). What is the point of this calculation?\n- Below Eq (5) you show diagrammatic notation and say that it represents the tensor $A$---but doesn't that tensor network graph correspond to the contraction in Eq (2)?\n- The discussion at the start of Sec 4 is difficult to follow. TNLM is never defined.\n- How exactly is the slice operator $T[i]$ defined? This is unclear below Eq (6). Is the 2nd or 3rd index kept fixed when slicing?\n- In Fig 3: Plot (a) shows that $f_\\theta(x^{(j)})$ is contracted with the tensor $M^{(j)}$. But then (b) and (c) seem to indicate that $M^{(j)}$ itself is a reshaped version (a matrix, rather than tensor) of a contraction that involves $f_\\theta(x^{(j)})$. Also, since TTML-Large and TTML-Tiny add additional operations like reshape, it's not immediately clear that they can even be expressed on the form in Eq (2). Again, if additional functions need to be added into Eq (2) to make the model expressible in that format, then it's not a TTLM.\n- In Fig 4: The caption refers to labels (RNNs-100, RNNs-200, etc) that don't appear in the figure.\n\n**W5.** The experiment results aren't represented properly. For example, you say that \"TTLM-Large obtains the best PPL among these models,\" but that's not true. Two of the LSTM-based methods achieve lower PPL in Table 1 (for PTB). For the PTB dataset, the PPL for TTLM-Large is bolded even though it's not the best number.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject