TextTN: Probabilistic Encoding of Language on Tensor Network
Authors: Peng Zhang
Jing Zhang
Xindian Ma
Siwei Rao
Guangjian Tian
Jun Wang
Venue:  (2021)
Keywords: Tensor Network
Language Representation
Natural Language Processing
Quantum Machine Learning
Entanglement Entropy

Abstract: As a novel model that bridges machine learning and quantum theory, tensor network (TN) has recently gained increasing attention and successful applications for processing natural images. However, for natural languages, it is unclear how to design a probabilistic encoding  architecture to efficiently and accurately learn and classify texts based on TN. This paper proposes a general two-step scheme of text classification based on Tensor Network, which is named as TextTN. TextTN first encodes the word vectors in a probabilistic space by a generative TN (word-GTN), and then classifies a text sentence using a discriminative TN (sentence-DTN).  Moreover, in sentence-DTN, its hyper-parameter (i.e., bond-dimension) can be analyzed and selected by the theoretical property of TextTN's expressive power. In experiments, our TextTN also obtains the state-of-the-art result on SST-5 sentiment classification task.

Under review as a conference paper at ICLR 2021
TEXTTN: PROBABILISTIC ENCODING OF LANGUAGE ON TENSOR NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
As a novel model that bridges machine learning and quantum theory, tensor net- work (TN) has recently gained increasing attention and successful applications for processing natural images. However, for natural languages, it is unclear how to design a probabilistic encoding architecture to efficiently and accurately learn and classify texts based on TN. This paper proposes general two-step scheme of text classification based on Tensor Network, which is named as TexITN. TextTN first encodes the word vectors in a probabilistic space by generative TN (word-GTN), and then classifies text sentence using a discriminative TN Cientence-DTNN Moreover, in sentence-DTN, its hyper-paramedd analyzed and selected by the theoretical property of TextTN's expressive power. In experiments, our TextTN also obtains the state-of-the-art result on SST-5 sentiment classification task.
(i.e., bond-dimension)
be
INTRODUCTION
Machine learning incorporating with the quantum mechanics forms a novel interdisciplinary field known quantum machine learning (Huggins et al.] [019 Ran al., 2020). Tensor network (TN) as a novel model has become prominent in the field of quantum machine learning (Biamonte) al. 2017). On the one hand, tensor network can be used as mathematical tool enhance the theoretical understanding of existing neural network methods (Levine et al. 2018 2019). On the other hand, based on tensor network, new machine learning algorithms have been proposed, e.g.. discriminative TN (DTN) (Stoudenmire & Schwab) 2016) for supervised tasks and generative TN (GTN) (Han et al. 2018) for unsupervised scenarios (Han et al., 2018). Based on the natural analogy between the quantum concepts (e.g., quantum many-body system (Levine et al. 2018)) and the image representation, many studies and applications are conducted for processing and learning natural pictures (Stoudenmire & Schwab 2016 [Sun et al. 2020 Liu et al. 2019). However, for natural languages, remains unclear how to design an efficient and effective TN approach, which can accurately learn and classify texts.
In the field of natural language processing (NLP), researchers have realized the analogy between the quantum many-body wave function and the word interactions (by the tensor product) in text sentence, and developed a quantum-inspired language representation (Zhang et al. 2018). Based on the quantum many-body physics and tensor decomposition techniques, ZZang et al. (2018) provided a mathematical understanding of existing convolution neural network (CNN) based text classification methods. Similarly, a tensor space language model (TSLM) has been built based network formulation (Zhang et al. 2019). This work shows that TSLM is more generalized language model compared with gram and recurrent neural network (RNN) based language models. In implementation, however, TSLM did not provide tensor network algorithm. The challenge lies in the high dimensionality of each word vector, which is much higher than the dimensionality of each pixel representation in image scenarios. After the tensor product of number of word vectors, the resulting high-order tensors will become computationally intractable.
the tensor
More recently, tensor network algorithm, namely uniform matrix product state (u-MPS) model, has been proposed for probabilistic modeling of a text sequence (Miller et al. 2020). u-MPS is evaluated on context-free language task, which uses synthetic data set. However, u-MPS has not been applied in a wolld NLP task, e.g., typical language modeling or text classification task. In addition, the expressive power of u-MPS has not been investigated. The expressive power of tensor

---

Under review as a conference paper at ICLR 2021
network is a fundamental property of various TNs and has been systematically studied for tensor network factorizations of multivariate probability distributions (Glasser et al. 2019). This motivates us to make use of the theoretical property of TN's expressive power, for developing a tensor network based probabilistic model for natural language representation and classification |
To build such text tensor network, we need to address two research problems in this paper. First, how to design a probabilistic encoding architecture to efficiently and effectively learn and classify the text. Second, how to analyse its expressive power, and make use of such analyses for more theoretical understanding and practical effectiveness of the text tensor network.
In this paper, we propose a novel tensor network architecture, named as TexITN. TextTN encodes each word vector in word-GTh, and classifies the sentence in a sentence-DTN. First, the proposed word-GTNs train a TN for each word and treat each element of word vector as node. In this manner, the word-GTNNs firstly map a high-dimensional word vector low-dimensional linear space by the tensor network operators. Then, the second layer tensor network, called as sentence-DTN, trains a TN for each sentence, by regarding the low-dimensional word vector obtained by word-GTNs as its input.
In TextTN, a sentence is represented by the tensor product among word vectors. Therefore, the interaction information among different word vectors and different dimensions are both modeled in TexTTN. Such interactions, encoded in the high-order weighted tensors, which represent high-order semantic space. In both word-GTNs and sentence-DTN, the high-order tensor can be solved by the tensor network, i.e., matrix product state model, which uses the idea of low-rank approximation that can conquer the exponential wall problem (Watson & Dunn 2010).
In sentence- DTN, the bond-dimension is an important hyper-parameter and reflects the expressive power of TexITN. In this paper, we analyze the upper and lower bounds of the bond-dimension. Particularly, its lower bound can be determined by the entanglement entropy, which can be considered as mcasurement of the communication information encoded in tensor network. A reference bond- dimension can be set as this lower bound, as we assume that larger value means an information redundancy and a smaller value indicates an insufficiency of the TN's expressive power. In the experiments, such a reference nond-dimension can achieve effective classification results, which indicates the TextTN's 'ss practical advantage in its potential to save paperrparrmmetr tuning efforts.
Moreover, the word interaction has been taken into account in sentence-Dii DDNN by the joint effect of different words for the later class predication by the loss functions. For the learning algorithm, we observe that different word positions have different weights in a sentence, so that the one- munfunntion (for a specific position) training in the original DTN is inappropriate. Therefore, we propose an all-function training process in the sentence- DTN to improve the stability of TexITN.
We have evaluated TexITN in four major text classification datasets (MR, Subj. CR and MPQA). The results show that TextTN outperforms convolutional neural network (CNN) on all the datasets. This departs from vision tasks where according the recent literature, tensor network has not been reported to outperform CNN (Kim 2014). In addition, based on the word vectors from the pre-trained model BERT, the TextTN has better results than the BERT model on SST-2 and SST-5 tasks, and the accuracy of BERT+ FERT+TextTTS is comparable with the state of the art (SOTA) result on SST-5 dataset.
2 BACKGROUND
We now provide the background of Matrix product States (MPS), a family of tensor networks. MPS (Schollwock) 2011) is also known the tensor train decomposition (Oseledets) 2011). Because of the low degree of freedom, the research based on MPS is developing rapidly. At present, the tensor network based on MPS can be roughly divided into two categories. One is the Generative Tensor Network (GTN) (Han et al. 2018 Sun et al. 2020), and the other one is the supervised tensor network (also named as Discriminative Tensor Network, DTN) (Stoudenmire Schwab) 2016). Then, we briefly describe existing GTN and DTN models for image classification tasks.
GTNS are used to model the joint probability distribution of given data. For a picture X with n pixels, each pixel is encoded into a two-dimensional vector x; (p:, 1 - pT) by a feature mapping from a
In this paper, focus the text classification task. However, the idea and formulation of our proposed approach are general and have potential in other NI.P tasks.
2

---

Under review as a conference paper at ICLR 2021
GTNs
DTN
2
Asy
As2
Ann
22
nn
As
A
As
As
A
Asp
W
W1
(X)
Xn
((X)
Figure 1: The schematic of tensor networks based on MPS. Left is GTNs and right is DTN.
pixel's value (Sun et al. 2020 where E {1,.... n and p; is the mapped probabilities of the pixel Ty. The representation of the picture X be give (xX) through the operator of tensor product between these vectors xy. A joint probability distribution of the picture X computed by the GTNs.
Px = w (X))2
where Py(X) represents the probability of picture X with respect to the category j (j (1..... m)). m is the number of the categories on an image classification dataset, and is the operator of tensor contraction. The MPS decomposition of the jth n-order weight tensor W, can be written
W Σ
A A
In the left of the Figure [ we give out an illustrative GTNs in two categories (i.e., m=2). Sec- ond or third order tensors As; (i E (I....,n)) are from the decomposition of the weight tensor W,. Each 'virtual' indicator ag (k fl...., - 1)) is the rank obtained from the tensor-train decomposition (Oseledets] 2011), and the "physical' indicator s, is the dimension of the pixel 2;.
DTN used to identify the class or label of picture and computes a conditional probability distribution X)).X). In DTN, the conditional probability distribution P(y)X) is computed as follows.
P(y|X) w (X).
(3)
where the (o+t))-ordr weight tensor W is decomposed into a MPS and is an extra order/index representing the label:
10 W Σ
1
(a]
where ( X) is vector and encodes the conditional probability distribution of outputting y given picture X. Eq. iss defined to classify the input X by choosing the label for which the value in the vector P(y] X) is largest. In practice. the MPS shown in the right of Figure is supervised learning model (Stoudenmire & Schwab 2016). and these rank values {az} are set to be equal the hyper-parameter (bond-dimension) TN.
TENSOR NETWORK FOR LANGUAGE ENCODING AND CLASSIFICATION
Problem setting Our goal is develop a tensor network architecture to encode and classify sentence of words in a TN's probabilistic manner. For sentence with n words (i.e, S=(w).... W,)).. the tensor representation 2019). where ui; are the word vectors. However, aforementioned in Introduction, because of the dimensional disaster, is infeasible to directly input such as a sentence representation into a tensor network. In order to solve this problem and still model the word interaction on high-order tensor (denoted as W'),
the sentence can written as (w) @... @ w; X ... @ w,, (Zhang et al.
formalize the TN model as follows:
P() w f(S) w (f(w)) @ f(w2)) ..... & f(w,,)
(5)
where f is operator encode a sentence in low-dimensional probabilistic space.
In our work, word-GTh are used to encode the word vectors into low-dimensional space, and then the new representation of word can be efficiently inputted to the tensor network. Specifically, the
3

---

Under review as a conference paper at ICLR 2021
function f in Eq 5 is embodied as word-GTN, (in Section B.1 and W is embodied sentence- DTN (in Section 3.2). In the process of text classification, the word-GTNs and sentence- Metennce-DTN are unified into new TN framework (TextTN), shown in Figure 2 Bond-dimension is important hyper-parameter which reflects the expressive power of 'TN model and influences the effectiveness of text classification. In Section 3.2 we propose reference bond-dimension that can be computed based on the entanglement entropy. Besides, to improve the stability of sentence-DTN, all-function learning on sentence-DiT Doonnce--TNN- proposed in Section 3.3
3.1 WORD ENCODING BASED ON WORD- WORD-GTNNS
We unfasten TextTN step by step in the sentence representation and classification problem. First, word-GTNs are used to encode word vectors to diwddmmnsional space, where m(m > 1) corresponds to the dimension of the low-dimensional space. In Figure 2 GTNs are used to encode each word in sentence, with shared parameters for each word. Firstly, each element of word embedding vector is mapped by feature mapping. Then, each word is represented as d-order tensor by calculating the tensor product of each element after feature mapping. For example, for word vector wn (01..... 0a)², the tensor product representation is shown as follows:
as an example, two word-
cos 5 sin T
COS I O4a) sin (5 0a)
((w,)
(6)
where @ is the tensor product, and of (w;) is the tensor product representation of word w, The feature mapping (i.e., cos (-) and sin (.)) is motivated by "spin" vectors encoded in quantum systems, which is usually used in TN (Stoudenmire & Schwab 2016 Han et al., 2018).
Then, two word-GTNs are used to encode the tensor ((w,; and get the probability distribution us i [I.....n} (see Figure 2). Specifically, for word us, each word-GTN can encode the word into a probability value j E (1,2]. In the following, es is represented by a 2-dimensional vector (Pi, Pi)"" which is calculated by:
P PS
(w:)) (ww;))
f(w,)
W
where is the operator of tensor contraction, W1 and W2 are two weight tensors to be learned in word-GTNs, and shared for all word vectors in a sentence. The probability condition of Pi+ is then achieved by the so coftmaz in practice.
word-GTNs are different from the existing GTNs shown in Background section. The word-GTNNs are used learning the new word representation (rather than text representation), and the existing GTNs are used to classify the pictures (not for pixels). Since word does not have class label, the output word-GTNs corresponds to the probabilities a low-dimensional feature space.
Regarding the dimension of low-dimensional space of word-GTNs. i.e, we will show that the upper bound of the bond dimension of TextTN is ln/ in Theorem 1 model will increase exponentially along with the increasing m (Khrulkov et al.) (2018). Therefore, the dimension should be set small value (e.g.. 2 in this section) to prevent too large expressive power. This observation is also supported by our experiments.
The expressive power of the
3.2 ANALYSIS OF BOND-DIMENSIO ON SENTENCE-DTY
After we use GTNss the new representation of sentence S can be written as (v) & @ v,). which is a n-order tensor. We now introduce a sentence- DTN classify sentences. This sentence- DTN is MPS which is from the tensor-train decomposition (Oseledets) (011)) of high-order tensor. By inputting the sentence vector obtained by word-GTNs, the sentenc-D'TV can formalized as:
P(y|S) = w (v) .... v..)
As, (1) As, (12) A's, (u:)... As, (v%)
which is a n+l-orddr MPS. As we introduced in Background, we can set the rank values as an equal value (bond-dimension), i.e., 01 =oy= =.. -
In other words, the bond dimension is set
4

---

Under review as a conference paper at ICLR 2021
sentenceDIIN (s-DTN)
01
an-2
an-1
As
1
Probahility Encoding
V1
2
nz
word-GINs (w-G7Ns)
(ww))
(ww)
( 2)
W1
W2
We-1
Wnn
Figure 2: The diagram about the model architecture of TextTN..
(r=az). Then, each element of W can be calculated as follows,
W S SI.... Sn.
Σ [ay=1}
Agt ... A 4 19,
A0n-1
The bond-dimension is the hyper-parameter which can reflect the expressive power of a tensor reference bond-dimension, which is an optimal parameter
network. We provide an approach setting reported in our experiments. In the following, we will derive the bound of the bond-dimension and show that lower bound is the reference recommended cand-dimensoon. This lower-bound, is calculated by the entanglement entropy, which will be described as follows.
Definition 1. For n+ I-order weight lensor W decomposed to matrix product states (MPS), X (A S r) singular values can be given out in the [ ] node of MPS. The entanglement entropy is calculated by the singular-value vector 9k as
E E=
E
gh log
(10)
where À is the number of non-zero singular values, log is the log to the base 2, and the normalization condition of the singular-value vector E R 1 satisfied.
Entangiement entropy measures the amount of communication information in TN (Levine et al. 2019), and we can derive the lower bound of the bond fond-dimension based on entanglement entropy, as shown in Theorem
Theorem 1. For the input (vi...., vn) of the sentence-DiN, the bond-dimension bounded as m mln/2] 2 bond-dimension 2 _
(11)
where the entanglement entropy of TexiTN, m the inputting dimension of sentence-DTN, the length of a sentence, and [] indicates rounding down.
Proof. The proof can be found in Appendix
B
According to Eq. IT the lower bound of bond- fond-dimension can be determined by the entanglement entropy E. If the bond-dimension is less than [ TextTTN can not model sufficient information measured by the entanglement entropy F. Compared with the lower bound [2 choosing larger value which satisfies Eq II will increase the number of parameters and the whole computational cost of the TexiNN, but only gain little information from the data. addition, the increase of the bond-dimension will increase the model complexity, possibly leading the overfitting problem. Therefore, we recommend the lower bound 2 as the reference bond-dimension.
About calculating the required entanglement entropy E, first need to sufficiently large dine-siieniion Since the initial bond -indedinensoon is large enough and the entanglement entropy finally converges, the E. measures the amount of information that the network can capture from

---

Under review as a conference paper at ICLR 2021
certain data set (Levine et al. 2019). After calculating the entanglement entropy E, according the proof of the Theorem we get the lower bound of din-edmnnsion shown as Eq which is the reference bond nond-dimension.
Moreover, we can derive the upper bound of the bond-dimension. According to the research (Krrulko ett al. 2018), the upper bound of the bond-dimension can be given out and m |n/221, where m is the inputting dimension of each node in MPS, and n is the number of the nodes, i.e., the length of sentence in TexTTN. If the bond-dimension is greater than m mln/2. then the network structure will not meet the constraints of Tensor-Tran decomposition (Oseledets) 2011) (Appendix) B). then TextTN will be rank deficient.
3.3 ALL-FUNCTION LEARNING ON SENTENCE-DTN
In the original DTN, the position of in Figure can be moved from As, to As. in the process of training. It means that any position be selected for the label bound (, leading different positions in the processes of training and prediction. The following Claim [ shows that the original DTN calculates conditional probability by putting the label bound in certain of the model, which corresponds to a certain certain probability function. However, the effects of putting the label bound in different positions should be taken into consideration.
variance for
the original DTN (Stoudenmire & Schwab 2016 computes the conditional probability P(y)S) by putting the label bound l in a certain position, which can be considered as a function from set Tof functions which compute all the possible conditional probabilities. Such a set defined as:
Claim 1. For an input S = {U1, U2..... ,Ua. Vah
(P(y)o U2...Uu).................. P(ylv1....... U; U2: Us+1..., U,. P(ylU1 In 1;
(12)
where: the location of segmentation for the imput S, and is also a position of label
Proof. The proof can be found in Appendix C
In the original DTN (Stoudenmire & Schwab 2016), the loss function is defined as L(W") Σ (S(f w'), based on the label bound in certain position, which only corresponds to certain probability function. If one function is learned, the different aspects probability distribution for the label bound in different position can not be considered. To solve this problem, we propose to model all functions in the set I'' in Eq. 12 with a new loss defined as follows:
L(W g) CE n
(f(())) ((1)) w
(13)
where CE is cross-entropy, y is the label that is represented by one-hot vector, N is the num- ber of samples, n is the length of the sentence, g is a parameter vector and gi is a value in the vector. (S),,, ) is one function from the function set T. The sentence-DTN with all-function learning is expected to achieve higher accuracy than the original DTN.
Finally, we give a Algorithm to describe our training method overall TextTN given a sentence S with words in the Algorithm 1.
4 EXPERIMENTS
In this section, we divide the experiment into two parts based on the theories presented above. The first set of experiments is to verify the effectiveness of TextTN. The second set of experiment is to verify that the effectiveness of the reference nond-dimension, which can be computed by Eq. []] based on the entanglement entropy for different datasets. Python implementation of the proposed algorithm is placed in Supplementary Material.
4.1 TEXT CLASSIFICATIO EVALUATION OF TEXTTN
Datasets and Benchmarks: Six text classification datasets are used in our experiments. MR (Pag
Lee 2004b): Movie reviews divided into positive and negative categories; CR (Hu & Liu

---

Under review as a conference paper at ICLR 2021
Algorithm Training Algorithm of TextTN
Input: 5 = {uri,..., Wn); ; wi E Rd (word embedding vectors), y E R (the label of S);
Output: W1. 2 and W (parameters tensors);
1: Initialize 11 and W2 ( for word-GTNs); W' (for sentence-DTNss
2: Initialized training: the reference bond-dimension for W obtained by the process.
3: repeat
4: Feature mapping: w, E - o(w)) E IRdx
5: Generate probability codings: v so (maaa(WW) ((w;)) W: o(w,))=), o2 E IR3;
6: Compute conditional probability; y w (V1,-.., vn y E R!:
7: Use Cross-Entropy method (CH) to calculate loss: min I. = CF( y):
8:
Perform backpropagation by L and update parameters W1, W2 and W
9:
until The loss L converges
Customer reviews set where the task is to predict positive or negative product reviews; Subj: Subjectivity dataset where the target is classify text being subjective objective;
2004):
MPQA (Wiebe et al. 2005): Opinion polarity detection subtask; SST-5 (Pang & Leel (2004a):
The movie reviews in the Stanford Sentiment Treebank, which is a fine-grained classification task (negative, somewhat negative, neutral, somewhat positive, positive); SST-2 (Socher et al. 2013):
Stanford Sentiment Treebank with binary classes. The evaluation metric is "Accuracy" of all tasks.
For CR. MR, MPQA, and Subj tasks, we use publicly word vectors (Word2vec) (Pennington et al. 2014 with 300 dimensions. For SST-2 and SST-5. the Bert-Large pre-trained model (Devlin et 2019 is used to obtain word vectors, and the dimension of the word vector from the BERT is 1024. More dataset statistics and experimental settings are given out in the Appendix D..I]
Table 1:
The comparison of experimental re- sults between TexTTN and other benchmarks. Accuracy is the evaluation metric.
Table 2:
The comparison between TexTTN and other complicated models based on pre-trained model.
Model
MR
CR
Subj
MPQA
Model
SST-5
SST-2
CNN
81.5
85.0
93.4
89.6
BCN+ELMo
54.7
DiSAN
84.8
94.2
90.1
Star- Transrraaformer
53
Capsule-B
82.3
85.1
93.8
FLOATER- large
96.7
SGC
75.9
CNN_Large
94.6
MPSAN
85.4
94.6
90.4
BERT
52.9
94.6
TexITN
82.2
85.7
95.3
90.4
BERT+TexIT]
95.3
Table 3: Ablation experiments. The comparative results between TexITN without all-function learning
(TextTN w/o all-function), TexTTN without word-GTNs (TexTTN w/o w-GTNs), and TextTN on four classification datasets (CR, Subj, MPQA and MR). The accuracy is adopted as the evaluation metric.
Model
MR
CR
Subj
MPQA
TextTN w/o w-GTNs
78.6
80.2
92.2
88.0
TexITN w/o all-function
80.5
84.1
94.4
89.8
TexTTN
82.2
85.7
95.3
90.4
Ten benchmarks are used to compare with TexITN. In Table I CNN (Kim 2014 4) is used for classifi- cation; DiSAN (Shen al. 2017 7) a self-attention based model; Capsule-B (Yang et al.
2018) is
capsule neural network; SGC (Wu et al. 2019) proposes graph neural network; MPSAN (Dai et al. 2020) designs a novel self-attention network. In Table 2. BCN+ELMo (Peters et al. 2018 is a deep bidirectional language model; Star- Transformer (uoo et al.
2019
is improved Transformer
network; BERT (Devlin et al. 2019). CNN Large (Baevski et al.
2019 and FLOATER (Liu et al.
2020) complicated neural network models based on the BERT
Large pre-trained model.
Experimental Results and Analysis: First, TextTN is experimented on four datasets. The results in Table I show that TextTN has better results than CNN (Kim) 2014) on MR (+0.7%), CR (+0.7%),
7

---

Under review as a conference paper at ICLR 2021
MR
M0:
CR
Sutg
4.25
(3)
00.
F13-
122
Rccurccyy
139
Accunecy Entanglement
Accuracy
Accurcty
Entanglement
Entanglement
277
50.
Enterglement
Engroyy
Entrody
Entrayy
Entropy
Aerd
On four classification tasks (MR, CR. MPQA and Subj), calculate the entanglement entropy and evaluate the accuracy of TexITN, respectively, with different bond-dimensions set from to 30.
Figure 3:
Subj (+1.9%) and MPQA (+0.8%). Moreover, compared with the state-of-the-art self-attention based model MPSAN Dai al. 2020), TextTN outperforms its results CR by 0.3%, Subj by 0.7%, and obtains a same result 90.4% o on MPQA.
After that, TextTN is experimented with the pre- pre-training word vectors from BERT on SST-2 and SST-5. The results are shown in Table In this experiment, BERT+TexUTN achieves the comparable result with the model ELECTRA (Clark et al. 2020), and obtains higher results than BERT (Devlin) itt al. 2019) by 0.7% on SST-2. For SST-5 dataset, BERT+TeX(TN outperforms the state-of-the-art results 54.7 by 0.1%, and is higher than BERT (Devlin al. 2019) by 1.9%.
In order to verify the effectiveness of all-function learning and word-GTNs, we perform ablation experiments shown as Table B First, when TextTN is without the all-function learning (TextTN w/o all-function), the classification accuracy declines to varying degrees on the four data sets. The accuracy values on MR, CR, Subj and MPQA drop by 1.7%, 1.6%, 0.9% and 0.6%, respectively. The results verify that the method of all-function learning for text classification is effective and useful.
In addition, in Table 3 we evaluate the effectiveness of word-GTNs by ablation experiments. TextTN w/o w-GTNs means that we use linear function of the word embeddings to obtain the low-dimensional word representations, which are inptted to the sentence-DTN, On the contrary, TexITN uses word-GTNs generate sentence representations. Compared with TexITN, the accuracy of TextTN w/o w-GTNs are largely reduced on four dataset. In particular, CR drops by 5.5%, MR, MPQA, and Subj decrease by 3.6%, 3.1%, and 2.4%, respectively. This shows that word-GTNs can effectively model word dependence in sentences and provide more effective semantic information for the classifier.
Finally, we analyze the impact of different word vector dimensions on the performance of TextTN, and the results are shown in the Appendix [..2 In addition, in the Section 3.1 the dimension m of probability encoding is a hyper-parameter set to 2. In order to verify this choice, we conduct a comparative experiment, and the results in Appendix illustrate ilaluusttte that compared with m 2, the accuracy of classification with m > 2 has not been improved, or even decreased.
4.2 EVALUATION OF THE REFERENCE BOND-DIMENSION OF TEXTTN
In Theorem 1, we show that the lower-bound of the bond-dimension can be derived by the entangle- ment entropy, and then consider such a lower bound can be the reference bond-dimension of Te In order to evaluate such reference First, we can compute the entanglement entropy through the initialized training of TextTN on text classification datasets (see Appendix DJ4 for details). After that, based on the entanglement entropy, the reference bond-dimension can computed.
value, we designed experiments as follows.
For MR, CR, MPQA and Subj datasets, their accuracy scores are highest when the bond-dimensions of sentence-DTN are 21,18,19 and 20, respectively. These values (i.c., 21,18,19 and 20), are actually the reference bond-dimensions, which are computed based on the entanglement entropy from the initialized training. Specifically. these values can be computed by Eq. 11 i.e., |24.40] = 21, [24.20 18, 2 19 and [24.3]] the accuracy values reach the highest point when the reference bond-dimenssons are computed as (MR),, 18(CR), 19(MPAA) and 20(Subj).
20, respectively. In Figure four charts show that
8

---

Under review as a conference paper at ICLR 2021
As shown in Figure by reducing and increasing the bond-dimension with respect its reference value, we also observe interesting results about the accuracy. For example, in the MR task, when bond dimension 21 accuracy is less than the accuracy when the bond-dimension is the reference value 21. If bond dimension 2 21, the accuracy begins to decline, but the updated entanglement entropy value (computed by the non-zero single values of sentence-DTN) remains stably. The same results are shown on the other three data sets, i.e., CR, MPQA and Subj.
In summary, the above results show the effectiveness of the reference bond-dimension. However, more rigorous analysis about the reference bond-dimension and the classification effectiveness of TextTN is still an open research question.
5 CONCLUSION AND FUTURE WORK
In this paper, propose text tensor network, named TexITN, which aims at encoding each word (in sentence) from a high dimensional vector to a low-dimensional vector by word-GTNs. After that, the new sentence representations classification. The proposed TextTN achieves better results than CNN, and even better experimental results than some neural network models, such HCapsNet. In addition, TexITN also achieves the experimental results that largely exceeds GTNs in all datasets. The entanglement entropy is defined in TexITN and helpful to identify the the reference bond-dimension which usually achieves the best accuracy in our experiments.
be inputted to the sentence-DTN carry out the sentence
In the future, would like to further investigate more theoretical evidence and understanding of using of the entanglement entropy to set an optimal bound dimension. In the practical point of view, we can apply TexTTN on more NLP tasks and test the effectiveness of both TextTN and its reference hyper- hyper-parameters. parameters.
REFERENCES
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5363-5372, 2019.
Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe, and Seth Lloyd. Quantum machine learning. Nature, (:9((67)) 2017.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders discriminators rather than generators. arXiv preprint arXiv:2003./0555 2020.
Biyun Dai, Jinlong Li, and Ruoyi Xu. Multiple positional self-attention network for text classification. In AAAL, pp. 7610-7617, 2020.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Ivan Glasser, Ryan Sweke, Nicola Pancotti. Jens Eisert, and Ignacio Cirac. Expressive power of tensor-network factorizations for probabilistic modeling. In Advances in Neural Information Processing Systems, pp. 1498-12 1510, 2019.
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star- transformer. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 1315-1325, 2019.
Zhao-Y Yu Han, Jun Wang, Heng Fan, Lei Wang, and Pan Zhang. Unsupervised generative modeling using matrix product states. Physical Review X, 8(3):03111,, 2018.

---

Under review as a conference paper at ICLR 2021
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Won Kim, Ron Kohavi, Johannes Gehrke, and William DuMouchel (eds.), Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, Washington, USA, August 22-25, 2004, pp. 168-177. ACM, 2004. doi: 10.1145/1014052.1011 77doi org. 10. 11 4044455501552111 101407
URL https
William Huggins, Piyush Patil, Bradley Mitchell, K Birgitta Whaley, and E Miles Stoudenmire. Towards quantum machine learning with tensor networks. Quantum Science and technology, 4(2): 024001, 2019.
Valentin Khrulkov, Alexander Novikov, and Ivan V. Oseledets. Expressive power of recurrent neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:://ppenreviee net forum di S1WRibb02)
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746-1751, 2014.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. http: 7/arxiv. otg/ abs/ 1412. 6980
Yoav Levine, David Yakira, Nadav Cohen, and Amnon Shashua. Deep learning and quantum entanglement: Fundamental connections with implications to network design. In 6th International Conference on Learning Representations, ICLR 2018. Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
Yoav Levine, Or Sharir, Nadav Cohen, and Amnon Shashua. Quantum entanglement in deep learning architectures. Physical review letters, 122(66)0 2019.
Ding Liu, Shi-Ju Ran, Peter Wittek, Cheng Peng. Raul Blázquez García, Gang Su, and Maciej Lewenstein. Machine learning by unitary tensor network of hierarchical tree structure. New Journal of Physics, 21(7):073059, 2019.
transformer with continuous dynamical model. arXiv preprint vrXiv:2003.0922: 2020. Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Learning to encode position for
Jacob Miller, Guillaume Rabusseau, and John Terilla, Tensor networks for language modeling. arXiv preprint (=XXi:20003.1033, 2020.
Alexander Novikov, Mikhail Trofimov, and Ivan Oseledets. Exponential machines. arXiv preprint 1rXiv:1605.03375, 2016.
Ivan Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295-
2017,
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summariza- tion based on minimum cuts. In Donia Scott, Walter Daelemans, and Marilyn A. Walker (eds.), Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 21-26 July, 2004, Barcelona, Spain, pp. 271-278. ACL, 2004a. doi: (0.3115/1218855511188889 URL
https: / www. . www.www.b..... wrg. hntho ogy hpwp:/044.0 0004/0041000 10357
Bo Pang and Lillian Lee. A sentimental education:
Sentiment analysis using subjectivity summariza- tion based on minimum cuts. In Donia Scott, Walter Daelemans, and Marilyn A. Walker (eds.), Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 21-26 July, 2004, Barcelona, Spain, pp. 271 ACI,, 2004b. doi: 12.1115/1118955.1118999 URL
https:/ //tw//wwwww awwweb.wb........... org/ wwt.aanthoooyyyooooo Ptw.oooyyy/4410000
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
10

---

Under review as a conference paper at ICLR 2021
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zett Deep contextualized word representations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computatioonal Linguistics: Human Language Technologies, Volume (Long Papers), pp. 2227-2237, 2018.
Shi-Ju Ran, Emanuele Tirrito, Cheng Peng, Xi Chen, G Su, and M Lewenstein. Tensor network contractions. LNP, 964, 2020.
U Schollwock. The density-matrix renormalization group in the age of matrix product states. Annals of Physics, 326(1):96-1)))) 2011.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan:
Directional self-attention network for mn//cnn-free language understanding. arXiv preprini arXiv:1709.04699 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 3, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, Special Interest Group of the ACL, pp. 1631-1642. ACL, 2013. URL httpp::/wwwt ac web org httpg///ahhhoooooooooo1o11
Miles Stoudenmire and David J Schwab. Supervised learning with quantum-insppred tensor networks. arXiv: Machine Learning. 2016.
Zheng-Zhe Sun, Cheng Peng, Ding Liu, Shi-Ju Ran, and Gang Su. Generative tensor network classification model for supervised machine learning. Physical Review B, 101(():075135, 2020.
Deborah K Watson and Martin Dunn. Rearranging the exponential wall for large n-body systems. Physical review letters, 105(2):02040,, 2010.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. language resources and evaluation, 39:165-210, 2005.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International Conference on Machine Learning, pp. 6861-6871, 2019.
Min Yang, Wei Zhao, Jianbo Ye, Zeyang Lei, Zhou Zhao, and Soufei Zhang. Investigating capsule networks with dynamic routing for text classification. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 1110-1199 2018
Lipeng Zhang, Peng Zhang, Xindian Ma, Shuqin Gu, Zhan Su, and Dawei Song. A generalized language model in tensor space. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. -450,7458, 2019.
Peng Zhang, Zhan Su, Lipeng Zhang, Benyou Wang, and Dawei Song. A quantum many-body wave function inspired language modeling approach. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 1303-1312, 2018.
A BASIC KNOWLEDGE ABOUT TENSOR NETWORK (TN)
We now describe some basic knowledge on tensor and tensor contraction, as well the matrix product states (MPS).
A.I TENSOR AND TENSOR CONTRACTION
A tensor can simply be thought of multi-dimensional array. The order of tensor is defined to the number of indexing entries in the array. For example, in Figure/p, the vector v with the index
i E m² is one order tensor, and the matrix A with two indexes (or nodes) i E m and j E is
2/ [m] also written i (1,2...,m.

---

Under review as a conference paper at ICLR 2021
two-order tensor. After that, n-order tensor is drawn in Figure with nodes, d1, and dn. where each node corresponds to an imput on TN.
Tensors have some basic operations. The most fundamental one is tensor contradiction, which means summing over same indices of two tensors. Two examples are presented here in order to illustrate the operation more directly. In Figure 4p, we show the process of matrix (2-order tensor) contracting with vector (1-order tensor). In Figure[pp. another contraction process between two matrix and B is demonstrated. And those high-order tensors, the tensor contraction operations between them follow the same rules with the above examples.
A.2 MATRIX PRODUCT STATE (MPS)
A Tensor Network weighted graph, where each node corresponds tensor. It has linear architecture with list of nodes and edges, and the order of the tensor is the degree of the node in the graph. The weight of each edge is referred 2016; Han et al. 2018)), MPS is kind of TN, which decomposed by a high-order tensor through Censor-Train decomposition (Oseledets) 2011). First, as shown in Figure 5p, the high-order tensor W is decomposed into multiple low-order tensors. Then, through tensor contraction, the low-order tensors represented by nodes can be contracted into MPS network Figure 5p. The formula is as follows:
its bond dimension. In (Novikov al.)
W
Σ
A
A
dy
RN-1
(14)
In the MPS, 0I, 02, .... ON the indexes of dind_dimensoons (also named TT-rank in some works), d, R" (i [N]). If the low-order tensors is not trimmed in the process of TT decomposition, that is, the full rank guaranteed, then 0 R 02 Rm² ,0[N/2] R RMLNN/ ....aN R". That is, the maximum bond-dimension is m [N/2]
Therefore, we avoid directly using the high-order tensor and Tensor-Train decomposition due to its exponentially increasing parameters. During the practical process, we do not construct a high- order tensor, nor do perform tensor decomposition. Instead, usually initialize an MPS architecture with random generated low-order tensors, optimizing them while training in order to better approximate feature distribution of data (which can be understood high-order tensor). Within the allowable deviation range, the amount of parameters will be dramatically reduced from m N m X r, and is the value of bond-dimension.
iE [n]
iE [m]
iE [1]
[x]
A
u
A
Av=u
Σ
Agp 4
J E [m]
Voctor (I-order tersor)
Matrix @orrtir tansor)
IE [n]
E [m
B
ki [4]
EE [E]
dg
dp-1
AB C
Σ, An Byx = Ci
W
Norderr tensor
(a)
(c)
Figure 4: The schematic diagram of tensors and tensor contraction
12

---

Under review as a conference paper at ICLR 2021
4
d2
dx-1
dy
W
d2
N-order temsor
A
A2
AN
Temsor-Train Train Decomposition
Tensor contraction
dy
A
Temsor-Train
d1
d2
dx
Decomposiion
022
d2
A
A2
AN
A
A2
AN
MPPS
(a)
(b)
Figure 5: The schematic diagram of Tensor- Train decomposition and Matrix Product State
class
class
class
9.(h)
92(h)
9x(0
$100
RiD
84(0
Pln/3]
Piuzjjs
(M,. Payzy
(". Va)
91
0
((()
0
0
((S)=wi0()
0
93
0
0
Figure 6: The diagram about the computing of singular values for the entanglement entropy. The left is the position separation sentence-DTNN (the position 0 Un//2) ), and the right is the representation of singular values in the process of SVD for centence-DTN.
B PROOF OF THEOREM
In TextTN,, the bond-dimension an important hyper-parameter which directly influences the expressive power of tensor network models (i.e., TexrTN). An reference bond-dimension can be given out based on the entanglement entropy which is defined Definition 1., Therefore, The Theorem I shows the relation between the entanglement entropy and the bond-dimension, and gives out the lone and the upper-bond bond of the hond-dimension. The proof of inequality [5 showed here.
Theorem 1. For the input of the sentence-Dre DDN {vi...., B, the bond- dimension is bounded
m |n/2| dime-dimnnsion |
(15)
where E is the entanglement entropy of TextTN, m the inputting dimension sentence-DTNy is the length of a sentence, and indicates rounding down.
Proof. As for the upper bound of the Eq. 15 as shown in Appendix A.2 sentence- DTN as MPS tensor network can be represented by decomposed n-order tensor. If rank is cut during
13

---

Under review as a conference paper at ICLR 2021
decomposition, the Suppose the formula of n-order tensor is:
(TT-rank) of sentence-DIN is the same the n-order tensor.
T (v) @ .... (n/2 @ (V|n/2]+1 @ .... & vn))
(16)
0y is the word probability encodings shown in Eq a m" tensor. From the matrix maximum rank theorem, the maximum rank in T appears the position [n/2], that is, the maximum rank is m |n/2|, Thus, the upper bound of bond-dimension in Sentence-DTN is also m |n/211
As for the lower bound of Eq. T5 the entanglement entropy E first computed by the first initializa- tion training. For the lower bond of bond cond-dimension, practically, the singular values between e |nn2] and D [m/2]++1 in Figure can be obtained in the training process of TextTN, when TextTN converges.
Assume that bond-dimension of TexITN is equal to k. The maximal entanglement entropy is computed when the singular values in TextTN are equal. Eq. 17 gives out the maximal entanglement entropy based fixed bond-dimension k.
&
1 k log k,
=
log
(17)
both sides the equation, doing the exponential calculations in base 2 on both sides of the equation, the new equation can be computed as follows,
2
(18)
When & singular values are not equal, the inequality can be written
k 2 2
(19)
To ensure that bond - dimension represented by k, is an integer, the inequality can rewritten as k 2 [2 where indicates rounding down.
Therefore, the inequality bond dimension 2 12 in Eq 15 can be established. Recall that the entanglement entropy E computed by Eq. 17 represents the maximal amount of information that TexITN can model. Eq. [9]iplisss that the maximal entanglement entropy E determines the lower bound of the cond-dimension of TexITN. For the upper bond, a parameter tensor W E Im" can be viewed matrix with the shape m² x m=-i The maximal rank can be obtained when the parameter tensor W is viewed as the matrix with the shape n/2] x m- m= In this case, the value of maximal rank can be equal to m [n/2], which is the upper bond of bond-dimension.
Therefore, the upper bond and lower bond of bone-dimension be obtained.
Theorem T has also been validated in practice. In the process of TextTN training, can compute the entanglement entropy of TextTN according to the Definition T When this training process converges, the maximal entanglement entropy obtained. In addition, TextTNs with different bond-dimensions are experimented. At the beginning, the classification accuracy of TextTN increases with the increasing of the bond-dimension. When the -indedimnnsoon increases to certain value, the accuracy of TextTN begins to decline slowly. Such a certain value is the reference bond-dimension and can be computed by _ when the training process converges.
C PROOF OF CLAIM 1
Claim For an input S = [V1, V2..... U }n) the original DTN (Stoudenmire & Schwab 2016 computes the conditional probability P(y)S) by putting the label bound in a certain position, which can be considered as certain function from a set T' of functions which compute all the possible conditional probabilities. Such set defined
(P(yloi ..... ... U(y)e..,PP))) ... Uúi U.+1..., ..., vn), ... P(y|v 1n v,))
(20)
where the location of segmentation for the input S. and is also position of label bound 1.
14

---

Under review as a conference paper at ICLR 2021
(S)
(2(
("(
(S)
(S)
(")))
$1
S2
S
f(S)
(a)
(b)
Figure 7: The diagram of all-function learning for tensor network. (a) The conditional probability functions for the output in different position Si(i (I,...,n)) (b) Getting an all-function ((5) by combining all functions (S) from different positions.
Proof. For an input S=(v),.... V. which is from output word-DTNs, a n-order tensor, i.e., the input of the sentence-DTN, is given b the operator of tensor product between vectors, v,, i E {I....,,A n). The parameter tensor W DTNN is (n + 1)-order tensor. There a mode (or an index) in tensor W to control the number of class in the classification task, which is I, As shown in Figure (a), in the processing of training, the index is moved from S1 to S.-
When the position of l is S1, the input is split parts, which I=(w)} and J=( ... ). the basic events are and J, the classification function (5) can be represented by condition probability follows,
(S) P() P1y V.... U,),))))
(21)
When the position of separation is 1<i<n, the input S is separated into two parts, which are [=(v,,..., d. and J= (U.4,..... v, the function f' is represented
(S(S) P(y De; 0i+1 v.).
(22)
When the position of separation is i=n - 1, the function (S) is denoted Eq. 23 similar to Eq. 21 and Eq. 22
f (S) Plylo U.
(23)
For different functions S'(S). they represent different condition probabilities. In the existing work (Stoudenmire & Schwab. 2016), i.e., original DTN, one function is used to classify the input S.
In order to model all functions in the process of training, and one function learning cannot model all conditional probabilities in the input S, we propose all-function learning method for text clas- sification, which is showed in Figure 7 (b). We adopt weighted summation for these functions ((.e. (SS), where g a parameter vector, gi an element from the vector g).
D EXPERIMENTAL RESULTS AND ANALYSIS
In this section, we first describe the datasets used in our experiments.
D.1 DATASETS
The statistics of six datasets are showed in Table 4 For CR, MR, MPQA and Subj datasets, they all have total file. Therefore, randomly select 10% of the file as the validation set and test set, respectively, and select 80% the file as the train set. The evaluating indicator of all text classification tasks is the "Accuracy".
For CR. MR, MPQA and Subj tasks, we implement our model with Pytorch- 1.20, and train them on Nvidia P40 GPU. As for learning method, and an exponentially decaying learning rate with a linear warm up. For SST-2 and SST-5 dataset, the initialized learning rate is 5e-5.
the Adam optimizer Kingma & Ba (2015)
15

---

Under review as a conference paper at ICLR 2021
Table 4: Statistics of all datasets. Metrics: Evaluation Metric, maximum sample length, N::
Dataset examples, Ntrain: training examples, Niett testing examples, Nde:: verification examples, [V|: vocabulary size, C: number of target categories.
Dataset
Metrics
N
Ntraón
Niest
Naev
[V]
C
CR
Accuracy
105
3775
CV
CV
CV
5340
2
MR
Accuracy
56
10662
CV
CV
CV
18765
2
MPQA
Accuracy
36
10606
CV
CV
CV
6246
Subj
Accuracy
120
10000
CV
CV
CV
21323
2
SST-2
Accuracy
65
~67K
~65K
1821
872
CV
2
SST-5
Accuracy
53
11855
8544
2210
1101
17836
435
Crampirner Drone
Figure 8: The entanglement entropp different training epochs on four datasets, including MR, CR, MPQA and Subj when the initialized training.
D.2 COMPARISON BETWEEN DIFFERENT DIMENSIONS OF PRE-TRAINED WORD VECTORS
As shown in Table 5 compare the accuracy of TextTN using word2vec (300-dimension) and Bert_ large 1024-dimension, pre- pre-trained word respetslesypyctivelly for nine text classification tasks (e.g., MR, CR and SST-5). The results show that the accuracy for six tasks will have a significantly increase as the dimension of word representations increases TexTTN. Especially for SST-5 and MR, the results of 1024-dimensional word embedding has over 5% increases compared with the results of 300-dimensional word embedding. Comparative experiments show that TextTN has better learning effects for high-dimensional word representations containing more semantic information.
Table 5: Experiments with varying input word embedding dimensions in six classification datasets
Model
MR
CR
Subj
MPQA
SST-5
SST-2
TextTN (dim-300)
82.2
95.3
91.4
TexiTN (dim-1024)
87.4
89.7
97.1
91.5
54.8
95.3
Table 6: Experiments with different dimensions of probability encoding Subj dataset.
Dimension 2
3
4
5
6
7
8
9
10
TexITN 95.3
95.0
94.5
94.2
94.6
94.4
94.7
94.6
94.4
D.3 A EXPERIMENT ABOUT THE DIMENSION OF PROBABILITY ENCODING
In Section 3.1.
we analyze that the dimension m of probability encoding can not exceed 2. this experiment, we evaluate the conclusion. As shown in Table we compare the accuracy of the TexITN with different probability encoding dimensions set from to 10. In addition, reported results are the average of 10 runs. The results illustrate that when
2. the accuracy of classification is 95.3, and when m > 2, the accuracy has a significant decrease. In particular, the accuracy only achieves 94.2 when dropping by 1.1 %. The Experimental results verify the effectiveness of m 2.
16

---

Under review as a conference paper at ICLR 2021
D.4 THE INITIALIZED TRAINING ON TEXTTN
In order to get the entanglement entropy shown as Eq in this paper) in the initialized training, we set 80 epochs training the TexITN on four text classification tasks (i.e., MR, CR, MPQA and Subj). In this process, the entanglement entropy, which is computed according to Definition I gradually converges a determinate value the increasing of training epoch, which shown as Figure
17


Reviews:
[{"review_id": "uaWQsMxRM1o", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary \nThe paper proposes a tensor network for text classification. There are two components: (i) word-GTNs convert word embeddings to m-d probability encoding vectors, and (ii) a sentence-DND takes the word probability encoding vectors as input, combining them using matrix product state (MPS). Experiments on several text classification datasets e.g. SST, CR, MPQA show that the proposed method outperforms existing ones when using word2vec and BERT word embeddings. \n\n### Contributions \nThe contributions are two-fold:\n\n* The paper proves an effective lower-bound for the bond-dimension. This is then confirmed by experimental results.\n\n* The used sentence-DND combines all possible locations inducing a distribution over the class set. Existing sentence-DNDs employs only location. \n\n###  Pros\nI do like theorem 1 about a lower-bound of the bond-dimension based on entanglement entropy. It is very interesting to see how experimental results support the theorem nicely (Figure 3). \n\nThe experimental results are a plus. \n\n### Cons\n\n* Clarity: \n   - Although the appendices help a lot, the paper isn't easy to read, especially to people who are not familiar with tensor. For instance, what is a \"tensor contraction\" (right after equation 7)? \n   - The paper also doesn't show the complexity. For examples, each d-dim word vector is represented by a \\phi(w) tensors, which should have 2^d parameters. In this case, what is the number of parameters of W in equation 7? what is the complexity of the whole model?\n   - How to compute entanglement entropy (Appendix D4) isn't provided.\n\n* Originality: In general, the paper doesn't seem to propose significant ideas. The two components are already used. \n   - In fact word-GTNs are just a classifier with a fix-sized input. These GTNs are used in computer vision. However, why don't just use a neural network mapping d-dim word embeddings to a distribution over m latent features. \n   - The extension of sentence-DTN seems to be trivial (actually I was wondering why no-one hadn't proposed that before). However, the extension should come with more complexity, which wasn't shown in the paper. \n\n* Motivation: The paper isn't convincing why using tensor networks is a good idea. It is unclear how tensors help the task (e.g. what is compositionality here?)\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "toVi7zvHNxk", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed TextTN that applied Tensor Network on text input. They effectively solved the high dimensionality problem and achieved the state of the art on multiple text classification problems. The paper also proved the bond dimension and showed experiment results to justify their theory.\n\nIn the experiments in table 1, was there a reason why you don't use the pretrained BERT embeddings? Can you compare with the BERT baseline as well?\n\nIn your BERT + TextTN experiment in table 2, do you finetune the pretrained BERT embeddings?\n\nCould you please analysis on the efficiency of your model? It could be a benefit of TextTN over the baseline models.\n\nThe results on some experiments over the baseline models are not very significant (by ~0.1). I am concerned how much of them comes from hyper-parameter tuning vs. model structure. It's possible that the tasks are not very hard so there are not too much headroom. Can you evaluate it on harder tasks such as other tasks in the GLUE benchmark?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Do9XIx618qZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After reading author replies:\nI would like to thank the authors to respond to my doubts on some of the results. But I decide to keep the review and the score, because Theorem 1 and Claim 1 are still not well explained. In particular, the explanation like \"if the 2nd inequality in Eq. 11 is violated, the network can not capture the amount of information measured by the entanglement entropy \" still looks like a conjecture or intuition rather than a mathematical statement. \n\n---------------------------------------------\n\nOriginality: High.\nThe proposed tensor network (TN) based text classification model looks new and interesting.\nIt consists of two parts: a word-level generative TN model, used to find concise representation of each word; and a sentence-level discriminative TN model, used to classify the sentence based on the outputs of the word-level TNs.\nBy using the word-level TN for input representation, the dimension explosion problem may be effectively avoided.\n\n\nClarity of model description: OK.\n\n\nClarity of training method: Low.\nThe training method of the overall model is not presented. Since there are two TNs, how to train the overall model may not be a trivial problem.\n\n\nClarity of analysis: Low.\nThe major problem of this paper is the analysis presented in Sec 3.2 and Sec 3.3.\nI understand that the authors may want to find some theoretical justification on how to choose the bond dimension in the sentence-level TN, as a function of the bond dimension of the word-level TN and the so called entanglement entropy.\nHowever, the result in Theorem 1 is incomprehensible. Only two inequalities are presented, without stating any condition or implication of the inequalities. Since both m and the bond dimension are hyper parameters, what does the 1st inequality mean? Is it a necessary condition on how to choose their values? What happens if the inequality is violated? Even the proof of this theorem does not answer these questions. Same doubts are on the 2nd inequality as well. Additionally, how does one even know what the entangle entropy of a model is before training the model?\n\nThe statement of Claim 1 is even more problematic. Not able to understand what it says.\n\n\nClarity of expeiremtal results: OK, but not clear enough.\nThe authors claim that when combined with BERT for word embedding, the proposed model can outperform the SOTA methods.\nHowever, there are several things that are not clear.\n1. It would be more fair to compared the combined model with BERT with some similar models that also use BERT for word embedding.\n2. It is claimed that the proposed method is better than word-GTN. But word-GTN is a unsupervised learning model. Why is it meaningful to make such a comparison?\n3. In the introduction, another TN based method \"TSLM\" is mentioned. Would it be more fair to compare the proposed method with TSLM, as both of them use TN for modeling?\n\n\nOverall, the proposed model looks interesting and shows some potential improvements over SOTA. However, the quality of the paper is degraded by the unclear claims and some questionable experimental results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DHzVxKHhyrE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nA tensor network model for text classification is introduced, which is constructed as the concatenation of a generative matrix product state (MPS) model for low-dimensional word embedding and a discriminative MPS model for classification. This model and different variants are assessed on multiple text classification datasets, with decent performance shown against a range of benchmarks.\n\nStrengths:\nThe TextTN model seems to be the first tensor network model applied to text classification. In that sense, the experimental results given are important for assessing the usefulness of tensor network models for real-world ML challenges, a question which has seen limited study so far.\n\nThe close correspondence shown between the accuracy in classification tasks and the entanglement entropy of the models (Figure 3) is interesting, and hints at the possibility of a compelling link between theoretical quantum many-body physics and practical considerations in ML.\n\nCritiques and Questions:\nThe word-GTN encoder (which converts high-dimensional word embeddings into low-dimensional inputs to the sentence-DTN) strikes me as being unnecessarily complicated, considering the small role it plays in the model. Given that this just outputs very low-dimensional word embeddings ($d=2$ in the paper), it would make sense to use a trainable word embedding here (or alternately a linear function of the original word embedding) in place of the word-GTN. On that note...\n\nThe \"TextTN w/o word-GTNs\" baseline in the ablation study (Table 3) seems rather misleading, as the paper text describes this as \"we directly average the word vectors of words in a sentence to obtain the sentence representation\". In other words, not only is the word-GTN removed, but the order of words is lost as well! I would request that this w/o word-GTN baseline to instead use a sentence-DTN whose inputs are low-dimensional vectors given by trainable low-dimensional word embeddings (or alternately a linear function of the word2vec word embeddings). This alternate baseline would make the TextTN significantly simpler (while still remaining a tensor network model), and would also bring it closer to the model of (Stoudenmire & Schwab 2016).\n\nRecommendation:\nI would recommend acceptance, owing to the new experimental evidence presented for the performance of tensor networks in NLP. However I do have some doubts about the encoding used for the model architecture, and request that the authors improve their ablation study to better justify the additional complexity coming from two separate linked matrix product state models.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the submission has promising components, the reviewers were not able to reach a consensus to recommend acceptance. The main concerns is that (1) theorem statements and assumptions are not clearly explained, and (2) the novelty of the approach is not made clear, and (3) there remain concerns on whether the experimental results are due to hyperparameter search or improvements due to the model.  
Decision: Reject