TextTN: Probabilistic Encoding of Language on Tensor Network

1 INTRODUCTION . Machine learning incorporating with the quantum mechanics forms a novel interdisciplinary field known as quantum machine learning ( Huggins et al. , 2019 ; Ran et al. , 2020 ) . Tensor network ( TN ) as a novel model has become prominent in the field of quantum machine learning ( Biamonte et al. , 2017 ) . On the one hand , tensor network can be used as mathematical tool to enhance the theoretical understanding of existing neural network methods ( Levine et al. , 2018 ; 2019 ) . On the other hand , based on tensor network , new machine learning algorithms have been proposed , e.g. , discriminative TN ( DTN ) ( Stoudenmire & Schwab , 2016 ) for supervised tasks and generative TN ( GTN ) ( Han et al. , 2018 ) for unsupervised scenarios ( Han et al. , 2018 ) . Based on the natural analogy between the quantum concepts ( e.g. , quantum many-body system ( Levine et al. , 2018 ) ) and the image representation , many studies and applications are conducted for processing and learning natural pictures ( Stoudenmire & Schwab , 2016 ; Sun et al. , 2020 ; Liu et al. , 2019 ) . However , for natural languages , it remains unclear how to design an efficient and effective TN approach , which can accurately learn and classify texts . In the field of natural language processing ( NLP ) , researchers have realized the analogy between the quantum many-body wave function and the word interactions ( by the tensor product ) in a text sentence , and developed a quantum-inspired language representation ( Zhang et al. , 2018 ) . Based on the quantum many-body physics and tensor decomposition techniques , Zhang et al . ( 2018 ) provided a mathematical understanding of existing convolution neural network ( CNN ) based text classification methods . Similarly , a tensor space language model ( TSLM ) has been built based on the tensor network formulation ( Zhang et al. , 2019 ) . This work shows that TSLM is a more generalized language model compared with n−gram and recurrent neural network ( RNN ) based language models . In implementation , however , TSLM did not provide a tensor network algorithm . The challenge lies in the high dimensionality of each word vector , which is much higher than the dimensionality of each pixel representation in image scenarios . After the tensor product of a number of word vectors , the resulting high-order tensors will become computationally intractable . More recently , a tensor network algorithm , namely uniform matrix product state ( u-MPS ) model , has been proposed for probabilistic modeling of a text sequence ( Miller et al. , 2020 ) . u-MPS is evaluated on a context-free language task , which uses an synthetic data set . However , u-MPS has not been applied in a real-world NLP task , e.g. , typical language modeling or text classification task . In addition , the expressive power of u-MPS has not been investigated . The expressive power of tensor network is a fundamental property of various TNs and has been systematically studied for tensor network factorizations of multivariate probability distributions ( Glasser et al. , 2019 ) . This motivates us to make use of the theoretical property of TN ’ s expressive power , for developing a tensor network based probabilistic model for natural language representation and classification 1 . To build such a text tensor network , we need to address two research problems in this paper . First , how to design a probabilistic encoding architecture to efficiently and effectively learn and classify the text . Second , how to analyse its expressive power , and make use of such analyses for more theoretical understanding and practical effectiveness of the text tensor network . In this paper , we propose a novel tensor network architecture , named as TextTN . TextTN encodes each word vector in word-GTNs and classifies the sentence in a sentence-DTN . First , the proposed word-GTNs train a TN for each word and treat each element of a word vector as a node . In this manner , the word-GTNs firstly map a high-dimensional word vector a low-dimensional linear space by the tensor network operators . Then , the second layer tensor network , called as sentence-DTN , trains a TN for each sentence , by regarding the low-dimensional word vector obtained by word-GTNs as its input . In TextTN , a sentence is represented by the tensor product among word vectors . Therefore , the interaction information among different word vectors and different dimensions are both modeled in TextTN . Such interactions , are encoded in the high-order weighted tensors , which represent a high-order semantic space . In both word-GTNs and sentence-DTN , the high-order tensor can be solved by the tensor network , i.e. , a matrix product state model , which uses the idea of low-rank approximation that can conquer the exponential wall problem ( Watson & Dunn , 2010 ) . In sentence-DTN , the bond-dimension is an important hyper-parameter and reflects the expressive power of TextTN . In this paper , we analyze the upper and lower bounds of the bond-dimension . Particularly , its lower bound can be determined by the entanglement entropy , which can be considered as a measurement of the communication information encoded in tensor network . A reference bonddimension can be set as this lower bound , as we assume that a larger value means an information redundancy and a smaller value indicates an insufficiency of the TN ’ s expressive power . In the experiments , such a reference bond-dimension can achieve effective classification results , which indicates the TextTN ’ s practical advantage in its potential to save hyper-parameter tuning efforts . Moreover , the word interaction has been taken into account in sentence-DTN by the joint effect of different words for the later class predication by the loss functions . For the learning algorithm , we observe that different word positions have different weights in a sentence , so that the one-function ( for a specific position ) training in the original DTN is inappropriate . Therefore , we propose an all-function training process in the sentence-DTN to improve the stability of TextTN . We have evaluated TextTN in four major text classification datasets ( MR , Subj , CR and MPQA ) . The results show that TextTN outperforms convolutional neural network ( CNN ) on all the datasets . This departs from vision tasks where according to the recent literature , a tensor network has not been reported to outperform CNN ( Kim , 2014 ) . In addition , based on the word vectors from the pre-trained model BERT , the TextTN has better results than the BERT model on SST-2 and SST-5 tasks , and the accuracy of BERT+TextTN is comparable with the state of the art ( SOTA ) result on SST-5 dataset . 2 BACKGROUND . We now provide the background of Matrix product States ( MPS ) , a family of tensor networks . MPS ( Schollwock , 2011 ) is also known as the tensor train decomposition ( Oseledets , 2011 ) . Because of the low degree of freedom , the research based on MPS is developing rapidly . At present , the tensor network based on MPS can be roughly divided into two categories . One is the Generative Tensor Network ( GTN ) ( Han et al. , 2018 ; Sun et al. , 2020 ) , and the other one is the supervised tensor network ( also named as Discriminative Tensor Network , DTN ) ( Stoudenmire & Schwab , 2016 ) . Then , we briefly describe existing GTN and DTN models for image classification tasks . GTNs are used to model the joint probability distribution of given data . For a picture X with n pixels , each pixel is encoded into a two-dimensional vector xi = ( pi , 1− pi ) T by a feature mapping from a 1In this paper , we focus on the text classification task . However , the idea and formulation of our proposed approach are general and have potential in other NLP tasks . pixel ’ s value ( Sun et al. , 2020 ) , where i ∈ { 1 , . . . , n } and pi is the mapped probabilities of the pixel xi . The representation of the picture X can be give out Φ ( X ) through the operator of tensor product between these vectors xi . A joint probability distribution of the picture X is computed by the GTNs . Pj ( X ) = |Wj • Φ ( X ) |2 ( 1 ) where Pj ( X ) represents the probability of a pictureX with respect to the category j ( j ∈ { 1 , . . . , m } ) , m is the number of the categories on an image classification dataset , and • is the operator of tensor contraction . The MPS decomposition of the jth n-order weight tensor Wj can be written as : Wj = ∑ { α } Aα1s1 A α1α2 s2 . . . A αn−1 sn ( 2 ) In the left of the Figure 1 , we give out an illustrative GTNs in two categories ( i.e. , m=2 ) . Second or third order tensors Asi ( i ∈ { 1 , . . . , n } ) are from the decomposition of the weight tensor Wj . Each ’ virtual ’ indicator αk ( k ∈ { 1 , . . . , n − 1 } ) is the rank obtained from the tensor-train decomposition ( Oseledets , 2011 ) , and the ’ physical ’ indicator si is the dimension of the pixel xi . DTN is used to identify the class or label of a picture and computes a conditional probability distribution P ( y|X ) . In DTN , the conditional probability distribution P ( y|X ) is computed as follows . P ( y|X ) = Wl • Φ ( X ) . ( 3 ) where the ( n+1 ) -order weight tensor Wl is decomposed into a MPS and l is an extra order/index representing the label : Wl = ∑ { α } Aα1s1 . . . A l ; αi−1αi si . . . A αn−1 sn ( 4 ) where P ( y|X ) is a vector and encodes the conditional probability distribution of outputting y given a picture X . Eq . 3 is defined to classify the input X by choosing the label for which the value in the vector P ( y|X ) is largest . In practice , the MPS shown in the right of Figure 1 is a supervised learning model ( Stoudenmire & Schwab , 2016 ) , and these rank values { αk } are set to be equal as the hyper-parameter ( bond-dimension ) in TN . 3 TENSOR NETWORK FOR LANGUAGE ENCODING AND CLASSIFICATION . Problem setting Our goal is to develop a tensor network architecture to encode and classify a sentence of words in a TN ’ s probabilistic manner . For a sentence with n words ( i.e , S= ( w1 , . . . , wn ) ) , the tensor representation of the sentence can be written as ( w1 ⊗ . . .⊗wi ⊗ . . .⊗wn ) ( Zhang et al. , 2019 ) , where wi are the word vectors . However , as aforementioned in Introduction , because of the dimensional disaster , it is infeasible to directly input such as a sentence representation into a tensor network . In order to solve this problem and still model the word interaction on a high-order tensor ( denoted as Wl ) , we can formalize the TN model as follows : P ( y|S ) = Wl • f ( S ) = Wl • ( f ( w1 ) ⊗ f ( w2 ) ⊗ . . .⊗ f ( wn ) ) ( 5 ) where f is an operator to encode a sentence in a low-dimensional probabilistic space . In our work , word-GTNs are used to encode the word vectors into a low-dimensional space , and then the new representation of word can be efficiently inputted to the tensor network . Specifically , the function f in Eq 5 is embodied as word-GTNs ( in Section 3.1 ) , and Wl is embodied as sentenceDTN ( in Section 3.2 ) . In the process of text classification , the word-GTNs and sentence-DTN are unified into a new TN framework ( TextTN ) , as shown in Figure 2 . Bond-dimension is an important hyper-parameter which reflects the expressive power of TN model and influences the effectiveness of text classification . In Section 3.2 , we propose a reference bond-dimension that can be computed based on the entanglement entropy . Besides , to improve the stability of sentence-DTN , all-function learning on sentence-DTN is proposed in Section 3.3 .

Summary: The paper proposes a tensor network for text classification. There are two components: (i) word-GTNs convert word embeddings to m-d probability encoding vectors, and (ii) a sentence-DND takes the word probability encoding vectors as input, combining them using matrix product state (MPS). Experiments on several text classification datasets e.g. SST, CR, MPQA show that the proposed method outperforms existing ones when using word2vec and BERT word embeddings.