Year: 2019

Abstract:
The formal understanding of deep learning has made great progress based on quantum many-body physics. For example, the entanglement entropy in quantum many-body systems can interpret the inductive bias of neural network and then guide the design of network structure and parameters for certain tasks. However, there are two unsolved problems in the current study of entanglement entropy, which limits its application potential. First, the theoretical analysis of entanglement entropy was only investigated in the representation of a single object (e.g., an image or a sentence), but has not been well studied in the matching of two objects (e.g., question-answering pairs). Second, the entanglement entropy can not be qualitatively calculated since the exponentially increasing dimension of the matching matrix. In this paper, we are trying to address these two problem by investigating the fundamental connections between the entanglement entropy and the attention matrix. We prove that by a mapping (via the trace operator) on the high-dimensional matching matrix, a low-dimensional attention matrix can be derived. Based on such a attention matrix, we can provide a feasible solution to the entanglement entropy that describes the correlation between the two objects in matching tasks. Inspired by the theoretical property of the entanglement entropy, we can design the network architecture adaptively in a typical text matching task, i.e., question-answering (QA) task. Our adaptive strategy sets a new state-of-the-art performance in TREC-QA task, with 2.9% absolute improvement over a recent model.

heading: 1 INTRODUCTION
text: Fundamental connections between neural network and quantum mechanics have been built (Carleo & Troyer, 2017; Levine et al., 2018; Cai & Liu, 2018). Neural networks are adopted to solve the quantum many-body problem (Carleo & Troyer, 2017; Cai & Liu, 2018), while quantum manybody function has been involved to explain the expressive ability of the neural networks (Levine et al., 2018). The formal understanding of deep neural network has made great progress based on quantum many-body physics. Levine et al. (2019) summarize the theoretical concepts and properties of quantum entanglement in different neural network architectures. Specifically, for recurrent networks, the Start-End separation rank as a method of measuring quantum entanglement, in order to quantitatively describe the depth effectiveness for modeling long-term memory capacity (Levine et al., 2017; 2019). For deep convolutional network, Levine et al. (2017; 2019), shows that the entanglement entropy can help us understand the inductive bias of the neural network. Such understanding also provide a guidance of how to control the inductive bias of the designed network, via the network parameters (e.g., channel numbers).
However, there are still two limitations in the current study (Levine et al., 2017; 2019; Zhang et al., 2018b). First, the quantum entanglement measure (i.e., the entanglement entropy) is considered as the indicator of the network’s expressive ability, but such an indicator only reflects the intricate correlation structures of each single input object (e.g., an image or a text). In other words, previous theoretical analyses are conducted only for the representation of a single object. However, for the
matching problem, the theoretical analysis should be extended, in order to model the correlation distributions between two objects (e.g., question-answering pairs).
On the other hand, the quantitative calculation of the entanglement entropy is also a problem. In theory, the entanglement entropy has a relation with the network parameters reflecting the inductive bias of the network. However, in practice, the entanglement entropy remains infeasible to calculate. This is due to the fact that the tensor product occurs in the quantum many-body function for representing the image and text (Levine et al., 2017; Zhang et al., 2018b), and the dimension of the matching matrix representing the quantum entanglement between two systems will increase exponentially. This leads to the difficulty in solving the entangled entropy by the singular value decomposition (SVD) of such a high-dimensional matching matrix.
In this paper, we are trying to address these two problem by investigating the fundamental connections between the entanglement entropy and the attention matrix. Under certain conditions and mappings, we demonstrate that a low-dimensional attention matrix can encode certain information (e.g., the trace of the block matrix) of the high-dimensional matching matrix. Then based on the lowdimensional attention matrix, we can quantitatively calculate entanglement entropy. Specifically, we can represent the high-dimensional matching matrix by block matrices, and then compute the trace for each block. We will show that the resulting matrix is equivalent to the attention matrix based on the cosine distance calculation. By the SVD decomposition of this matrix, the singular values are obtained to solve the entanglement entropy quantitatively.
In our work, inspired by (Levine et al., 2018), we can adaptively design convolutional network based on the calculable entanglement entropy. Specifically, for the more complex the inputs (i.e., the longrange correlations), more kernels should be assigned in the relatively-deeper layers, while more kernels should be assigned in the relatively-shallower layers for short-range correlations. Intuitively, in text matching task like question answering system, the short-range correlations can refer to some simple question-answering pairs with many common words between question and answer sentences, which can be matched locally by some overlapping features (e.g., the statistics of a single word or nearby word combinations like N-gram). While long-range correlations refer to the questionanswering pairs with less common words which their effective matching may need higher-level semantic information extracted from a global context. This strategy significantly improves the final performance on two typical text matching tasks called TREC-QA and YAHOO-QA. In particular,our model sets a new state-of-the-art performance in TREC-QA, with 2.9% absolute improvements.

heading: 2 BACKGROUND AND BASIC NOTATIONS
text: Since our work is mainly for the text matching task of a sentence pair, we briefly introduce a recent Quantum Many-body Wave Function inspired Language Modeling ( QMWF-LM) (Zhang et al., 2018b). In QMWF-LM, a state can be represented by a Dirac notation |w〉, which can be considered as a column vector w. For better understanding, in our paper, we will use a vector w to represent a Dirac vector |w〉. A word can be described by a state vector as follows.
w = α1e1 + . . .+ αmem (1)
where {e1, . . . , em} is a set of base vectors. Each αi(i ∈ 1, . . . ,m) is a probability amplitude and∑m i α 2 i = 1. In practice, w can be the distributive representation or the one-hot representation for the word w. A sentence can be represented by tensor product ⊗ 1 among words in a sentence. For the representation of a sentence S with n words, it can be written as follows,
s = w1 ⊗ . . .⊗wn (2) where {w1 . . .wn} are the word which is defined in E.q. 1. In order to establish the correlation between two sentences (VQ with a number (saying a) of words in the question and VA with b words in the answer), we can have the following equation. The state of the two sentences system can be defined as:
ψ = ma∑ i=1 mb∑ j=1 Tijφ Q i ⊗ φ A j (3)
1e.g., e1 = (1, 0)T , e2 = (0, 1)T , e1 ⊗ e2 = (0, 1, 0, 0)T
where the state vector ψ is a composite system between two sentences (i.e., VQ and VA), φ Q i is a high-dimensional (ma,m is the dimension of word vector) base vector for the sentence VQ, φ Q i also is a high-dimensional (mb) base vector for the sentence VA. Especially, Ti,j is the element for the high-dimensional (ma by mb) matching matrix T .

heading: 3 A FEASIBLE METHOD FOR QUANTIFYING ENTANGLEMENT ENTROPY
text: In this section, firstly, we use entanglement entropy to measure the quantum entanglement describing the correlation between two subsystems (e.g.,a sentence pair). Secondly, we prove the equivalence between the quantum entanglement and the attention matrix under certain conditions. Finally, based on the attention matrix, the entanglement entropy is quantitatively calculated, which facilitates us to design the neural network architecture through the overall inductive bias.

heading: 3.1 ENTANGLEMENT ENTROPY IN TEXT MATCHING
text: Entanglement entropy is used to represent correlations between two subsystems in deep neural network (Levine et al., 2018; 2019). Specifically, it requires a matrix to model all the quantum entanglement of the two subsystems, and then the matrix is decomposed to obtain the singular values, which often correspond to the important information hidden in the matrix, and the importance is positively correlated with the singular value size. Finally, based on the singular values, the entanglement entropy representing the degree of entanglement between the two subsystems can be obtained. In the matching task, we use quantum many-body wave functions (Zhang et al., 2018b) as the basic language representation to get the entanglement entropy between sentence pairs.
In matching tasks (e.g., Q&A task), a question-answering pair can be considered as two subsystems, VQ = {wQ1 , . . . ,wQa }, VA = {wA1 , . . . ,wAb }. VQ ∈ HQ with dimension ma and VA ∈ HA with dimension mb, respectively. m is the dimension of word vector. The composite system ψS of VQ and VA, which can be written as (Levine et al., 2018; Cohen & Shashua, 2016):
ψS = ma∑ i=1 mb∑ j=1 Ti,jφ Q i ⊗ φ A j (4)
where φQi (i ∈ HQ) and φAj (j ∈ HA) are basis vectors of VQ and VA, respectively. The dimensions of basis vectors about VQ and VA are ma and mb, respectively. All the entries Ti,j shows the correlation between φi and φj . When i = 1, j is taken from 1 to mb, φ Q 1 represents the basis vector of i = 1 in the tensor space HQ (each base vector represents a certain combined semantics of all words), {φQj }m a
j=1 represents all base vectors in the tensor spaceHQ, T1,j represents the corresponding probability amplitude of the base vector φQ1 of the VQ subsystem combined with the base vector φAj of the VA subsystem. The matching matrix T can obtain all probability amplitude distributions of i from 1 to ma in parallel.
The matching matrix T contains all the probability amplitude distributions between the two subsystem basis vectors. Singular value decomposition (SVD) (Schollwoeck & White, 2013) on the matching matrix T can be written:
T = r∑ i=1 λiϕQi ·ϕ T Ai (5)
where ϕQi and ϕAj are r vectors in new bases for H Q and HA, respectively. λ1 ≥ . . . ≥ λr > 0 are the singular values, which often correspond to the important information hidden in the matrix, and the importance is positively correlated with the singular value size, the size of the singular value represents the weight of the current basis vector of the two subsystems. λi is also understood as probability amplitude, ∑ i λ 2 i = 1. λi represents the weight of principal semantic bases between VQ and VA, and reflects the weight of effective semantics to some extent (since it is larger than zero). Entanglement entropy (Yo, 2015) is computed as follows on the matching matrix T ,
S = − r∑
i=1
|λi|2ln|λi|2 (6)
where S ∈ (0, ln (r)), Smax = ln (r), r is the number of non-zero singular values, and the number of r is called Schmidt number (Shi et al., 2006; Levine et al., 2018). The entanglement entropy describes the correlation between two systems, and the Schmidt number indicates the upper bound of the network expression ability and can assert the role of channel numbers of each layer in the overall inductive bias (Levine et al., 2018). Due to this dimensional catastrophe 2, it is infeasible to calculate entanglement entropy and Schmidt number. This leads us to practically apply inductive bias is limited in the experiment. However, in matching tasks, an attention matrix often is used to describe correlation between two systems. Based on the above ideas, we propose a claim to calculate the entanglement entropy.
Claim 1. Assume the compound relation of word vectors in VQ (or VA) are considered in the case of 1-order, and it just is considered for the compound relation between each word vector in VQ and each word vector in VA. An attention matrix can be received by computing the trace of block matrices of the matching matrix T .
Proof. Assume the compound relation of word vectors in VQ(or VA) are not consider (e.g., 1-order), the dimensions of HQ and HA in E.q. 4 are a×m (am) and b×m (bm), respectively. E.q. 4 can be rewritten as follow:
ψS = am∑ i=1 bm∑ j=1 Ti,jφ Q i ⊗ φ A j (7)
where φQi is the base vector from a group of complete base vector space φ Q 1 , . . . ,φ Q am, and φ A j is the base vector from a group of complete base vector space φA1 , . . . ,φ A bm. The T is a matching matrix of am × bm. Divide the matching matrix T into a × b block matrices {Pij ∈ Rm×m}(i ∈ [a]3; j ∈ [b]), which can be written as Pij = wQi ⊗ wAj . Each element Eij of the attention matrix E is equal to the inner product of two word vectors which are from the sets VQ and VA, which is Eij = w Q i ·wAj T (i ∈ [a]; j ∈ [b]). There is a mathematical connection between Pij andEij , which can be written as: Eij = trace(Pij) (8)
The relation between the matching matrix T and the attention matrixE has been shown as the Fig.1.
2Dimensional catastrophe is mainly reflected in two aspects. First, in Eq.4, the matrix T (T ∈ Rm a×mb ) has dimensional catastrophes. Second, even if the SVD is decomposed on the matrix T , its number of singular values still has dimensional catastrophe (i.e., number of singular values ≤ min(ma,mb)).
3[a] represents a set {1, 2, 3, . . . , a}
Thus, we use the attention matrix E instead of the matching matrix T to quantify the entanglement entropy. In our Conjecture 1, we provide a generalized proof of the mathematical relationship between the attention matrix and the entanglement properties. In the experiments, we use the cosine distance about two vectors to replace the inner product actually. In the next subsection, we will describe our Algorithm.

heading: 3.2 NETWORK DESIGN BASED ON ENTANGLEMENT ENTROPY
text: Recall that the sentence pair contained query sentence and answer sentence can be represented through two sets, which are VQ and VA, respectively. where VQ = {wQ1 , . . . ,wQa }, VA = {wA1 , . . . ,wAb }, and a+ b = n. Based on the Claim 1, the attention matrix between of VQ and VA can be computed by using cosine similarity in experiment,
Eij = wQi ·wAj
‖wQi ‖ × ‖wAj ‖ (9)
the attention matrix E ∈ Ra×b can be decomposed by SVD. After that, the entanglement entropy can is calculated through E.q., 6 on attention matrix E. while the upper bound of entanglement entropy Smax is obtained based on the number of singular values. The normalized value of the entanglement entropy is used in practice, it can be written as follows.
D = (Smax − S)/Smax. (10)
We calculate all data sample differences D, and then find the median Dm of the difference D. The samples with D greater than Dm are defined as the group with the short-range correlations, while the samples less than Dm are defined as the group with long-range correlations. Based on the connection between deep learning and quantum entanglement (Levine et al., 2018), i.e., deep learning describes the correlation required for deep learning tasks, we can adaptively design convolutional network by the correlation (e.g., long/short-range correlation). For the more complex the input (i.e., the long-range correlation), more kernels should be assigned in the relatively-deeper layers, while more kernels should be assigned in the relatively-shallower layers for short-range correlations. In our opinion, quantitatively calculating entanglement entropy by attention matrix can not only help us adaptively design the network architecture, but also provides a new perspective for us to understand the attention matrix in matching tasks.

heading: 4 EXPERIMENTS
text: Question Answering aims to rank the answers from a candidate answer pool for a given question. The ranking is usually based on two parts, namely, representation part for a single sentence and matching part between these two representations.
Datasets We validated our model through the TREC-QA (Wang et al., 2007) and YAHOO-QA (Tay et al., 2017) datasets in the matching task. TREC-QA dataset contains two training sets, namely TRAIN and TRAIN-ALL. We use TRAIN-ALL, which is larger and contains noisier questionanswering pairs, in order to verify the robustness of the proposed model. YAHOO-QA dataset is collected from yahoo answers for community-based question answering. The answers are generally longer than those in TREC-QA. As introduced in this paper (Tay et al., 2017), we select the QA pairs containing questions in which the token number of its corresponding answers is more than 5 and less than 59 after removing non-alphanumberic characters. For each question, we construct negative samples by ranking the top 4 answers from the whole answer sentences according to BM25.
Parameters Setting We utilize the Adam (Bengio & LeCun, 2015) optimizer with learning rate 0.001 and use the best model obtained in the dev dataset for evaluation in the test set. The batch size and L2 regularization are set to 64 and 0.00001, respectively. The 50-dimension word embeddings are trained by word2vec (Burges et al., 2013) on English Wikimedia dump, in which the Out-ofVocabulary words are randomly initialized by a uniform distribution in the range of (−0.25, 0.25).

heading: 4.1 EVALUATION METRICS
text: TREC-QA is evaluated by MAP(mean average precision), MRR(mean reciprocal rank), while YAHOO-QA is evaluated in term of P1 (precision at one) and MRR since it targets more on the first ranking item.

heading: 4.2 CONFIGURATIONS FOR OUR MODEL
text: In this paper, we adopt the quantum many-body wave functions (Zhang et al., 2018b) as the basic representation components, to represent both question and answers sentence as two subsystems i.e. Q and A respectively. Since the whole system with question-answering pairs is composed by these two subsystems, the entanglement entropy is adopted to quantify correlations between subsystems.
Generally speaking, there are typically two kinds of correlations in question answer task, namely long-range correlation and short-range correlation. The former refers to more complex matching with consideration of the global sentence. The latter refers to the cases that there are already overlapping words or phrases between the question sentence and the answer sentence, which can be matched locally at the lexical and syntactic level. As discussed in our algorithm, the entanglement entropy is a quantitative metric to distinguish how much semantically entangled the question and answer are. Thus we could divide question-answering pairs into two sub-datasets, i.e. long-range sub-dataset and short-range sub-dataset. Then, we can design an adaptive setting for kernels for different sub-datasets. For long-range sub-dataset, we should choose the more kernels in deep layers (namely deep-more-kernels mode). while for short-range sub-dataset, we should choose the more kernels in the shallow layers (namely shallow-more-kernels mode). At last, we combine the evaluation scores of the long/short range sub-datasets as the data for the whole dataset to be evaluated.
We set a parameter r from following modes for an input layer and three-layer CNN network:
(1) shallow-more-kernels: [m,3r,2r,m] (2) deep-more-kernels:[m,2r,3r,m]
m is a hyper-parameter for the kernel count of the last CNN layer, which is set as dimension of word embedding for guaranteeing the same parameter scale between these two modes. The parameter count for both configurations is identical: m · 3r + 3r · 2r + 2r ·m = 6r2 + 5mr. Shallow-morekernels mode has more kernels in the lower layers while the deep-more-kernels mode has more kernels in higher layers. The code will be open-sourced in 4.
Comparative Setup To validate the above adaptive setting for parameters, we design two comparative fixed settings without dividing the dataset. Fixed setting 1: the deep-more-kernels parameter mode is used for the whole dataset. Fixed setting 2: the shallow-more-kernels mode is applied to the
4https://github.com/anonymous/anonymous.git
whole dataset. Except for the setting for the kernel numbers and dataset paritition, Other parameters in both adaptive and fixed settings are the same, for a fair comparison.

heading: 4.3 RELEVANT METHODS FOR COMPARISON
text: QLM (Sordoni et al., 2013) adopts the density matrix of quantum mechanics to model the dependencies between words such as query words and documents.
NNQLM-II (Zhang et al., 2018a) is a end-to-end quantum-like language model for question answering. It adopts a convolution neural network over the multiplication between two density matrices.
QMWF-LM (Zhang et al., 2018b) is based on the quantum many-body wave function representation and a further convolution neural network for an approximate tensor decomposition.
CNM (Li et al., 2019) utilizes the probability properties of the density matrix, indirectly measures the distance between two density matrices through a trainable measurement operation, and uses training data to provide flexible matching scores.
Our model is also compared to a series of CNN and RNN models (Rao et al., 2016; Tayyar Madabushi et al., 2018; Tay et al., 2017; 2018; Kamath et al., 2019). To the best of our knowledge, Question Classification + RNN + Pre-Attention (Kamath et al., 2019) is the state of art in the raw version of TREC-QA. 5.

heading: 4.4 EXPERIMENTAL RESULTS
text: Tab. 1 reports the results on TREC-QA dataset and YAHOO-QA dataset. The first group shows a comparison of between our model with four quantum-inspired language models. For TREC-QA, our model significantly outperforms QLM by 20.3% on MAP and 19.8% on MRR, respectively. Compared with NNQLM-II and QMWF-LM, CNM, it also reflects the superiority of our model.
5https://aclweb.org/aclwiki/Question Answering (State of the art)
For YAHOO-QA, our model is significantly better than QLM 23.5% and 18.5% in P@1 and MRR, significantly better than NNQLM-II 16.4% and 11.6% in P@1 and MRR, and better than QMWFLM 5.5% and 4.4% in P@1 and MRR. In summary, compared with three quantum-inspired language models, our model shows good experimental performance on both tasks.
In the second group, our model with adaptive setting for kernels is compared with other two fixed settings, i.e., deep-more-kernels fixed for the whole dataset and shallow-more-kernels for the whole dataset (see Sec. 4.2 for details). For TREC-QA, the results show that both MAP and MRR are at least 15% better than them. For YAHOO-QA, the results show that both P@1 and MRR are at least 20% better than the comparative setup. In summary, the results reflect the effectiveness of the adaptive settings of parameters for two divided data-subsets.
In the third group, we compare our model with a number of CNN-based models and RNN-based models (Rao et al., 2016; Tayyar Madabushi et al., 2018; Tay et al., 2017; 2018). Multi-Perspective CNN (Rao et al., 2016) and Multi-Perspective CNN +PairwiseRank + Question classification (Tayyar Madabushi et al., 2018) model the QA task as a ranking task and proposes a pairwise ranking method that can directly utilize the existing point-by-point neural network model as a basic component. HD-LSTM (Tay et al., 2017) and CTRN (Tay et al., 2018), Question Classification + RNN + Pre-Attention (Kamath et al., 2019) propose an RNN-based approach to match question-answering pairs. It should be emphasized that our model is better than Question Classification + RNN + PreAttention (Kamath et al., 2019), which is the state of art in the raw version of TREC-QA. For TRECQA and YAHOO-QA, our models are better than the results (Rao et al., 2016; Tayyar Madabushi et al., 2018; Tay et al., 2017; 2018; Kamath et al., 2019) reported in the original paper.

heading: 4.5 DISCUSSION AND ANALYSIS
text: Analysis of Adaptive Settings In Fig. 2, we investigate the difference between the adaptive setting in our model and two fixed settings introduced. Recall that in our model, we first identify the correlation between sentence pairs based on the attention matrix and entanglement entropy. Then, according to the correlation, the question-answering pairs could be divided into two sub-dataset, i.e, long-range sub-dataset and short-range datasets. Therefore, we can adaptive set the kernels for different sub-dataset. On the other hand, the two fixed settings apply the same parameter setup (deep-more-kernels or shallow-more-kernels) for the whole dataset.
As we can see from Fig. 2 TREC-QA, in our model, with the increase of the parameter r, MAP, MRR and P@1 show an upward trend. For the fixed settings, i.e., deep-more-kernels for the whole dataset and shallow-more-kernels for the whole dataset, it does not show better performance as r changes, and even show a downward trend. This suggests that our adaptive kernels settings can well capture the language matching characteristics of the question-answering pairs, and the alternative approach to the entangle entropy as the measurement for the language correlations, work well for dividing the dataset into two subsets, reflecting different ranges of language correlations.
Influence of Channels in Convolution In Fig. 2, as r grows, the MRR and MAP of TREC-QA and YAHOO-QA increases. We select the optimal r in a range [16, 32, 64, 96, 128]. For different datasets, we set a same number of channels to obtain the final performance. In our model, for the TREC-QA and YAHOO-QA, when the parameter r is set to 128, the experimental results as shown in Tab. 1 can be obtained.
Efficiency Analysis we utilize adaptive settings of kernels for different sub-datasets. The efficiency relies on the deep convolution neural network. In our experiment, for TREC-QA, the training epoch is set to be 100, while for YAHOO-QA, after training 30 epochs, we will obtain the results.

heading: 5 CONCLUSIONS AND FUTURE WORK
text: In this paper, we aim to extend previous studies on the theoretical and practical use of the concept of quantum entanglement in neural network. We demonstrate the connection between the entanglement entropy and the attention matrix in text matching tasks. This allows us to quantitatively calculate the entanglement entropy, which helps us adaptively setting network structures and parameters, and achieve effective performance on two typical QA tasks. In future work, we will investigate the entanglement entropy under high-order conditions and in other networks, e.g., recurrent networks.

heading: A ATTENTION MATRIX AND ENTANGLEMENT ENTROPY
text: Given a sentence pair with word embedding {wi ∈ Rm}(i ∈ [n]), which can be represented using two sets, i.e., VQ = {wQ1 , . . . ,wQa }, VA = {wA1 , . . . ,wAb }. Conjecture 1. Assume the compound relation of word vectors in VA are considered in the case of n-order, and it just is considered for the compound relation between each word vector in VQ and b words vectors in VA. The attention matrix can be received by computing the trace of block matrices of matrix T .
Proof. Firstly, the mathematical relationship between the attention matrix and the quantum entanglement matrix is proved under the 1-order condition in 3.1.
Assume the compound relation of word vectors in VA are considered in the 2-order, the dimension ofHQ is a×m (am) andHA is C2b×m2, i.e., C2bm2 (C2b is a Combination number6). E.q. 4 can be rewritten as follow:
ψS = am∑ i=1 C2bm 2∑ j=1 Ti,jφ Q i ⊗ φ A j (11)
where {φQi }ami=1 and {φAj } C2bm 2
j=1 are base vectors forHA andHQ. The T is a martix of am×C2bm2. Divide the T into a× C2b block matrices {Pij ∈ Rm×m
2}(i ∈ [a]; j ∈ [C2b ]), which can be written as Pij = w Q i ⊗wAj ,wAj (wAj ∈ Rm 2
) represents the compound semantics of any two-word vectors in VA. Matrix Pij can be regarded as a 3-order tensor Pt1t2t3 (Pt1t2t3 ∈ Rm×m×m), each element Eij (i ∈ [a]; j ∈ [C2b ]) of the attention matrix E is equal to the sum of the 3-order tensor Pt1t2t3 diagonal elements, i.e., Eij =< w Q t1 ,w A t2 · w A t3 > (t1 ∈ [a], t2 6= t3 and t2, t3 ∈ [b]). there is a mathematical connection between Pt1t2t3 and Eij , which can be written as:
Eij = trace(Pij) (12)
Thus, we use the attention matrix E instead of T to quantify the entanglement entropy.
Secondly, assume the compound relation of word vectors in VQ(or VA) are considered in the k-order (k < b), the dimension ofHQ is a×m (am) andHA is Ckb×mk (Ckbmk). E.q. 4 can be rewritten as follow:
ψS = am∑ i=1 Ckb m k∑ j=1 Ti,jφ Q i ⊗ φ A j (13)
6Cmn represents the number of combinations of m elements taken from n different elements(m ≤ n)
where {φQi }ami=1 and {φAj } Ckb m k
j=1 are base vectors forHA andHQ. The T is a matrix of am×Ckbmk. Divide the T into a×Ckb block matrices {Pij ∈ Rm×m
k}(i ∈ [a]; j ∈ [Ckb ]), which can be written as Pij = w Q i ⊗wAj , wAj (wAj ∈ Rm k
) represents the compound semantics of any k word vectors in VA. Matrix Pij can be regarded as a k-order tensor Pt1t2...tk (Pt1t2...tk ∈ Rm×m×···×m), each element Eij (i ∈ [a]; j ∈ [Ckb ]) of the attention matrix E is equal to the trace of the k-order tensor, i.e., Eij =< w Q t1 ,w A t2 ·w A t3 . . .w A tk > (t1 ∈ [a], t2 6= t3 6= · · · 6= tk and t2, t3, . . . tk ∈ [b]), which can be written as: Eij = trace(Pij) (14)
Thus, we use the attention matrix E instead of T to quantify the entanglement entropy. Thirdly, assume the compound relation of word vectors in VQ(or VA) are considered in the (k + 1)-order (k < b), the dimension of HQ is a×m (am) and HA is Ck+1b ×mk+1 (C k+1 b m
k+1). E.q. 4 can be rewritten as follow:
ψS = am∑ i=1 Ck+1b m k+1∑ j=1 Ti,jφ Q i ⊗ φ A j (15)
where {φQi }ami=1 and {φAj } Ck+1b m k+1
j=1 are base vectors for HA and HQ. The T is a matrix of am × Ck+1b m k+1. Divide the T into a × Ck+1b block matrices {Pij ∈ Rm×m k+1}(i ∈ [a]; j ∈ [Ck+1b ]), which can be written as Pij = w Q i ⊗wAj , wAj (wAj ∈ Rm k+1
) represents the compound semantics of any (k+1) word vectors in VA. Matrix Pij can be regarded as a (k+1)-order tensor Pt1t2...t(k+1) (Pt1t2...t(k+1) ∈ Rm×m×···×m), each element Eij (i ∈ [a]; j ∈ [C k+1 b ]) of the attention matrix E is equal to the trace of the (k + 1)-order tensor, i.e., Eij =< w Q t1 ,w A t2 · w A t3 . . .w A t(k+1) > (t1 ∈ [a], t2 6= t3 6= · · · 6= t(k+1) and t2, t3, . . . t(k+1) ∈ [b]), which can be written as:
Eij = trace(Pij) (16)
Thus, we use the attention matrix E instead of T to quantify the entanglement entropy.
In summary, the equivalence relationship between the attention matrixE and the entanglement properties matrix T is satisfied under the 1-order condition, the 2-order condition, the k-order condition, and the (k+1)-order condition, which proves that the relationship is established. That is, attention and entanglement properties are equivalent under specific premise.

Review summaries:
In this paper, a method for leveraging entanglement entropy for understanding attention matrices is proposed. Specifically, the paper aims at solving two problems: 1) to study the theoretical analysis of entanglement entropy for the matching of two objects (question-answering pairs), and (2) to qualitatively calculate the matching matrix. The introduced approach is based on fundamental connections between the entanglement entropy and the attention matrix. The main goal of the paper is to show that a low-dimensional attention matrix can be derived from a high-dimensional matching matrix. Results are shown for a text matching task on two datasets (TREC-QA, YAHOO-QA).