Leveraging Entanglement Entropy for Deep Understanding of  Attention Matrix in Text Matching

1 INTRODUCTION . Fundamental connections between neural network and quantum mechanics have been built ( Carleo & Troyer , 2017 ; Levine et al. , 2018 ; Cai & Liu , 2018 ) . Neural networks are adopted to solve the quantum many-body problem ( Carleo & Troyer , 2017 ; Cai & Liu , 2018 ) , while quantum manybody function has been involved to explain the expressive ability of the neural networks ( Levine et al. , 2018 ) . The formal understanding of deep neural network has made great progress based on quantum many-body physics . Levine et al . ( 2019 ) summarize the theoretical concepts and properties of quantum entanglement in different neural network architectures . Specifically , for recurrent networks , the Start-End separation rank as a method of measuring quantum entanglement , in order to quantitatively describe the depth effectiveness for modeling long-term memory capacity ( Levine et al. , 2017 ; 2019 ) . For deep convolutional network , Levine et al . ( 2017 ; 2019 ) , shows that the entanglement entropy can help us understand the inductive bias of the neural network . Such understanding also provide a guidance of how to control the inductive bias of the designed network , via the network parameters ( e.g. , channel numbers ) . However , there are still two limitations in the current study ( Levine et al. , 2017 ; 2019 ; Zhang et al. , 2018b ) . First , the quantum entanglement measure ( i.e. , the entanglement entropy ) is considered as the indicator of the network ’ s expressive ability , but such an indicator only reflects the intricate correlation structures of each single input object ( e.g. , an image or a text ) . In other words , previous theoretical analyses are conducted only for the representation of a single object . However , for the matching problem , the theoretical analysis should be extended , in order to model the correlation distributions between two objects ( e.g. , question-answering pairs ) . On the other hand , the quantitative calculation of the entanglement entropy is also a problem . In theory , the entanglement entropy has a relation with the network parameters reflecting the inductive bias of the network . However , in practice , the entanglement entropy remains infeasible to calculate . This is due to the fact that the tensor product occurs in the quantum many-body function for representing the image and text ( Levine et al. , 2017 ; Zhang et al. , 2018b ) , and the dimension of the matching matrix representing the quantum entanglement between two systems will increase exponentially . This leads to the difficulty in solving the entangled entropy by the singular value decomposition ( SVD ) of such a high-dimensional matching matrix . In this paper , we are trying to address these two problem by investigating the fundamental connections between the entanglement entropy and the attention matrix . Under certain conditions and mappings , we demonstrate that a low-dimensional attention matrix can encode certain information ( e.g. , the trace of the block matrix ) of the high-dimensional matching matrix . Then based on the lowdimensional attention matrix , we can quantitatively calculate entanglement entropy . Specifically , we can represent the high-dimensional matching matrix by block matrices , and then compute the trace for each block . We will show that the resulting matrix is equivalent to the attention matrix based on the cosine distance calculation . By the SVD decomposition of this matrix , the singular values are obtained to solve the entanglement entropy quantitatively . In our work , inspired by ( Levine et al. , 2018 ) , we can adaptively design convolutional network based on the calculable entanglement entropy . Specifically , for the more complex the inputs ( i.e. , the longrange correlations ) , more kernels should be assigned in the relatively-deeper layers , while more kernels should be assigned in the relatively-shallower layers for short-range correlations . Intuitively , in text matching task like question answering system , the short-range correlations can refer to some simple question-answering pairs with many common words between question and answer sentences , which can be matched locally by some overlapping features ( e.g. , the statistics of a single word or nearby word combinations like N-gram ) . While long-range correlations refer to the questionanswering pairs with less common words which their effective matching may need higher-level semantic information extracted from a global context . This strategy significantly improves the final performance on two typical text matching tasks called TREC-QA and YAHOO-QA . In particular , our model sets a new state-of-the-art performance in TREC-QA , with 2.9 % absolute improvements . 2 BACKGROUND AND BASIC NOTATIONS . Since our work is mainly for the text matching task of a sentence pair , we briefly introduce a recent Quantum Many-body Wave Function inspired Language Modeling ( QMWF-LM ) ( Zhang et al. , 2018b ) . In QMWF-LM , a state can be represented by a Dirac notation |w〉 , which can be considered as a column vector w. For better understanding , in our paper , we will use a vector w to represent a Dirac vector |w〉 . A word can be described by a state vector as follows . w = α1e1 + . . .+ αmem ( 1 ) where { e1 , . . . , em } is a set of base vectors . Each αi ( i ∈ 1 , . . . , m ) is a probability amplitude and∑m i α 2 i = 1 . In practice , w can be the distributive representation or the one-hot representation for the word w. A sentence can be represented by tensor product ⊗ 1 among words in a sentence . For the representation of a sentence S with n words , it can be written as follows , s = w1 ⊗ . . .⊗wn ( 2 ) where { w1 . . .wn } are the word which is defined in E.q . 1 . In order to establish the correlation between two sentences ( VQ with a number ( saying a ) of words in the question and VA with b words in the answer ) , we can have the following equation . The state of the two sentences system can be defined as : ψ = ma∑ i=1 mb∑ j=1 Tijφ Q i ⊗ φ A j ( 3 ) 1e.g. , e1 = ( 1 , 0 ) T , e2 = ( 0 , 1 ) T , e1 ⊗ e2 = ( 0 , 1 , 0 , 0 ) T where the state vector ψ is a composite system between two sentences ( i.e. , VQ and VA ) , φ Q i is a high-dimensional ( ma , m is the dimension of word vector ) base vector for the sentence VQ , φ Q i also is a high-dimensional ( mb ) base vector for the sentence VA . Especially , Ti , j is the element for the high-dimensional ( ma by mb ) matching matrix T . 3 A FEASIBLE METHOD FOR QUANTIFYING ENTANGLEMENT ENTROPY . In this section , firstly , we use entanglement entropy to measure the quantum entanglement describing the correlation between two subsystems ( e.g. , a sentence pair ) . Secondly , we prove the equivalence between the quantum entanglement and the attention matrix under certain conditions . Finally , based on the attention matrix , the entanglement entropy is quantitatively calculated , which facilitates us to design the neural network architecture through the overall inductive bias . 3.1 ENTANGLEMENT ENTROPY IN TEXT MATCHING . Entanglement entropy is used to represent correlations between two subsystems in deep neural network ( Levine et al. , 2018 ; 2019 ) . Specifically , it requires a matrix to model all the quantum entanglement of the two subsystems , and then the matrix is decomposed to obtain the singular values , which often correspond to the important information hidden in the matrix , and the importance is positively correlated with the singular value size . Finally , based on the singular values , the entanglement entropy representing the degree of entanglement between the two subsystems can be obtained . In the matching task , we use quantum many-body wave functions ( Zhang et al. , 2018b ) as the basic language representation to get the entanglement entropy between sentence pairs . In matching tasks ( e.g. , Q & A task ) , a question-answering pair can be considered as two subsystems , VQ = { wQ1 , . . . , wQa } , VA = { wA1 , . . . , wAb } . VQ ∈ HQ with dimension ma and VA ∈ HA with dimension mb , respectively . m is the dimension of word vector . The composite system ψS of VQ and VA , which can be written as ( Levine et al. , 2018 ; Cohen & Shashua , 2016 ) : ψS = ma∑ i=1 mb∑ j=1 Ti , jφ Q i ⊗ φ A j ( 4 ) where φQi ( i ∈ HQ ) and φAj ( j ∈ HA ) are basis vectors of VQ and VA , respectively . The dimensions of basis vectors about VQ and VA are ma and mb , respectively . All the entries Ti , j shows the correlation between φi and φj . When i = 1 , j is taken from 1 to mb , φ Q 1 represents the basis vector of i = 1 in the tensor space HQ ( each base vector represents a certain combined semantics of all words ) , { φQj } m a j=1 represents all base vectors in the tensor spaceHQ , T1 , j represents the corresponding probability amplitude of the base vector φQ1 of the VQ subsystem combined with the base vector φAj of the VA subsystem . The matching matrix T can obtain all probability amplitude distributions of i from 1 to ma in parallel . The matching matrix T contains all the probability amplitude distributions between the two subsystem basis vectors . Singular value decomposition ( SVD ) ( Schollwoeck & White , 2013 ) on the matching matrix T can be written : T = r∑ i=1 λiϕQi ·ϕ T Ai ( 5 ) where ϕQi and ϕAj are r vectors in new bases for H Q and HA , respectively . λ1 ≥ . . . ≥ λr > 0 are the singular values , which often correspond to the important information hidden in the matrix , and the importance is positively correlated with the singular value size , the size of the singular value represents the weight of the current basis vector of the two subsystems . λi is also understood as probability amplitude , ∑ i λ 2 i = 1. λi represents the weight of principal semantic bases between VQ and VA , and reflects the weight of effective semantics to some extent ( since it is larger than zero ) . Entanglement entropy ( Yo , 2015 ) is computed as follows on the matching matrix T , S = − r∑ i=1 |λi|2ln|λi|2 ( 6 ) where S ∈ ( 0 , ln ( r ) ) , Smax = ln ( r ) , r is the number of non-zero singular values , and the number of r is called Schmidt number ( Shi et al. , 2006 ; Levine et al. , 2018 ) . The entanglement entropy describes the correlation between two systems , and the Schmidt number indicates the upper bound of the network expression ability and can assert the role of channel numbers of each layer in the overall inductive bias ( Levine et al. , 2018 ) . Due to this dimensional catastrophe 2 , it is infeasible to calculate entanglement entropy and Schmidt number . This leads us to practically apply inductive bias is limited in the experiment . However , in matching tasks , an attention matrix often is used to describe correlation between two systems . Based on the above ideas , we propose a claim to calculate the entanglement entropy . Claim 1 . Assume the compound relation of word vectors in VQ ( or VA ) are considered in the case of 1-order , and it just is considered for the compound relation between each word vector in VQ and each word vector in VA. An attention matrix can be received by computing the trace of block matrices of the matching matrix T . Proof . Assume the compound relation of word vectors in VQ ( or VA ) are not consider ( e.g. , 1-order ) , the dimensions of HQ and HA in E.q . 4 are a×m ( am ) and b×m ( bm ) , respectively . E.q . 4 can be rewritten as follow : ψS = am∑ i=1 bm∑ j=1 Ti , jφ Q i ⊗ φ A j ( 7 ) where φQi is the base vector from a group of complete base vector space φ Q 1 , . . . , φ Q am , and φ A j is the base vector from a group of complete base vector space φA1 , . . . , φ A bm . The T is a matching matrix of am × bm . Divide the matching matrix T into a × b block matrices { Pij ∈ Rm×m } ( i ∈ [ a ] 3 ; j ∈ [ b ] ) , which can be written as Pij = wQi ⊗ wAj . Each element Eij of the attention matrix E is equal to the inner product of two word vectors which are from the sets VQ and VA , which is Eij = w Q i ·wAj T ( i ∈ [ a ] ; j ∈ [ b ] ) . There is a mathematical connection between Pij andEij , which can be written as : Eij = trace ( Pij ) ( 8 ) The relation between the matching matrix T and the attention matrixE has been shown as the Fig.1 . 2Dimensional catastrophe is mainly reflected in two aspects . First , in Eq.4 , the matrix T ( T ∈ Rm a×mb ) has dimensional catastrophes . Second , even if the SVD is decomposed on the matrix T , its number of singular values still has dimensional catastrophe ( i.e. , number of singular values ≤ min ( ma , mb ) ) . 3 [ a ] represents a set { 1 , 2 , 3 , . . . , a } Thus , we use the attention matrix E instead of the matching matrix T to quantify the entanglement entropy . In our Conjecture 1 , we provide a generalized proof of the mathematical relationship between the attention matrix and the entanglement properties . In the experiments , we use the cosine distance about two vectors to replace the inner product actually . In the next subsection , we will describe our Algorithm .

Summary: In this paper, a method for leveraging entanglement entropy for understanding attention matrices is proposed. Specifically, the paper aims at solving two problems: 1) to study the theoretical analysis of entanglement entropy for the matching of two objects (question-answering pairs), and (2) to qualitatively calculate the matching matrix. The introduced approach is based on fundamental connections between the entanglement entropy and the attention matrix. The main goal of the paper is to show that a low-dimensional attention matrix can be derived from a high-dimensional matching matrix. Results are shown for a text matching task on two datasets (TREC-QA, YAHOO-QA).