Leveraging Entanglement Entropy for Deep Understanding of  Attention Matrix in Text Matching
Authors: Peng Zhang
XiaoLiu Mao
XinDian Ma
BenYou Wang
Jing Zhang
Jun Wang
DaWei Song
Venue:  (2020)
Keywords: Quantum entanglement entropy
Attention Matrix

Abstract: The formal understanding of deep learning has made great progress based on quantum many-body physics.  For example, the entanglement entropy in quantum many-body systems can interpret the inductive bias of neural network and then guide the design of network structure and parameters for certain tasks. However, there are two unsolved problems in the current study of entanglement entropy, which limits its application potential. First, the theoretical benefits of entanglement entropy was only investigated in the representation of a single object (e.g., an image or a sentence), but has not been well studied in the matching of two objects (e.g., question-answering pairs). Second,  the entanglement entropy can not be qualitatively calculated since  the exponentially increasing dimension of the matching matrix. In this paper, we are trying to address these two problem by investigating the fundamental connections between the entanglement entropy and the attention matrix. We prove that by a mapping (via the trace operator) on the high-dimensional matching matrix,  a low-dimensional attention matrix can be derived. Based on such a attention matrix, we can provide a feasible solution to the entanglement entropy that describes the correlation between the two objects in matching tasks. Inspired by the theoretical property of the entanglement entropy, we can design the network architecture adaptively in a typical text matching task, i.e., question-answering task.

Under review as a conference paper at ICLR 2020
LEVERAGING ENTANGLEMENT ENTROPY FOR DEEP UNDERSTANDING OF ATTENTION MATRIX IN TEXT MATCHING
Anonymous authors Paper under double-blind review
ABSTRACT
The formal understanding of deep learning has made great progress based on quantum many-body physics. For example, the entanglement entropy in quantum many-body systems can interpret the inductive bias of neural network and then guide the design of network structure and parameters for certain tasks. However, there two unsolved problems in the current study entanglement entropy, which limits its applica- tion potential. First, the theoretical analysis of entanglement entropy was only investigated in the representation of a single object (e.g., an image sentence), but has been well studied the matching of two objects (e.g., question-answeriin entropy can not be qualitatively calculated since the exponentially in- creasing dimension of the matching matrix. In this paper, we are trying address these two problem by investigating the fundamental connec- tions between the entanglement entropy and the attention matrix. We prove that by mapping (via the trace operator) on the high-dimensional matching matrix, a low-dimensional attention matrix can be derived. Based on such a attention matrix, we can provide a feasible solution to the entanglement entropy that describes the correlation between the two objects matching tasks. Inspired by the theoretical property of the en- tanglement entropy, we can design the network architecture adaptively in typical text matching task, i.e., question-answering (QA) task. Our adaptive strategy sets new state-of-the-art performance in TREC-QA task, with 2.9% absolute improvement over a recent model.
pairs). Second, the entanglement
INTRODUCTION
Fundamental connections between neural network and quantum mechanies have been built (ar- lco & Troyer 2017 Levine et al.) 2018 Cai & Liu| 2018). Neural networks are adopted to solve the quantum many-body problem (Carleo & Troyer 2017 Cai & Liu 2018), while quantum many- body function has been involved to explain the expressive ability of the neural networks (Levine) al. 2018).. The formal understanding of deep neural network has made great progress based on quantum many-body physics. Levine et al. (2019) summarize the theoretical concepts and proper- ties of quantum entanglement different neural network architectures. Specifically, for recurrent networks, the Start-End separation rank as a method of measuring quantum entanglement, in order to quantitatively describe the depth effectiveness for modeling long-term memory capacity (Levine) eet al., 2017 2019). For deep convolutional network, Levine et al. (2017) 2019), shows that the entanglement entropy can help us understand the inductive bias of the neural network. Such under- standing also provide guidance of how control the inductive bias of the designed network, via the network parameters (c.g., channel numbers).
However, there are still two limitations in the current study (Levine et al. 2017: 2019 Zhang al. 2018b). First, the quantum entanglement measure (i.e., the entanglement entropy) is considered as the indicator of the network's expressive ability, but such indicator only reflects the intricate correlation structures of each single input object (e.g., an image or a text). In other words, previous theoretical analyses conducted only for the representation of single object. However, for the

---

Under review as a conference paper at ICLR 2020
matching problem, the theoretical analysis should be extended, in order to model the correlation distributions between two objects (e.g., question-answeringg -nswering pairs).
On the other hand, the quantitative calculation of the entanglement entropy is also problem. In the- ory, the entanglement entropy has a relation with the network parameters reflecting the inductive bias of the network. However, in practice, the entanglement entropy remains infeasible calculate. This is due to the fact that the tensor product occurs in the quantum many-body function for representing the image and text (Levine al. 2017 Zhang et al. 2018b), and the dimension of the matching ma- trix representing the quantum entanglement between two systems will increase exponentially. This leads to the difficulty in solving the entangled entropy by the singular value decomposition (SVD) of such sigh-dimensional matching matrix.
In this paper, we are trying to address these two problem by investigating the fundamental con- nections between the entanglement entropy and the attention matrix. Under certain conditions and mappings, we demonstrate that dindeeesioonal attention matrix can encode certain information (e.g., the trace of the block matrix) of the nigh-dimensional matching matrix. Then based on the low- dimensional attention matrix, we can quantitatively calculate entanglement entropy. Specifically, we can represent the high-dimensional matching matrix by block matrices, and then compute the trace for each block. We will show that the resulting matrix is equivalent to the attention matrix based on the cosine distance calculation. By the SVD decomposition of this matrix, the singular values are obtained to solve the entanglement entropy quantitatively.
In our work, inspired by (Levine al.) 2018), we can adaptively design convolutional network based on the calculable entanglement entropy. Specifically, for the more complex the inputs (i.e., the long- range correlations), more kernels should be assigned in the relatively-deepe layers, while more kernels should be assigned in the relatively-shallower layers for short-range correlations. Intuitively, in text matching task like question answering system, the short-range correlations can refer some simple question-answenee which can be matched locally by some overlapping features (e.g., the statistics of a single word or nearby word combinations like N-gram). While long-range correlations refer to the question- answering pairs with less common words which their effective matching may need higher-level semantic information extracted from global context. This strategy significantly improves the final performance on two typical text matching tasks called TREC-QA and YAHOO-QA. In particular, model sets new -tate-of-lue the-art performance in TREC-QA, with 2.9% absolute improvements.
pairs with many common words between question and answer sentences,
2 BACKGROUND AND BASIC NOTATIONS
Since our work is mainly for the text matching task of sentence pair, we briefly introduce a recent Quantum Many-body Wave Function inspired Language Modeling QMWF-LMM (Zhang et al.) 2018b). In QMWF-LM, a state can be represented by a Dirac notation |wr), which can be considered as a column vector un. For better understanding. in our paper, we will use a vector u to represent a Dirac vector (w). A word can be described by a state vector as follows.
a1e1 +... 2m
where fe1,..., Cm } is a set of base vectors. Each as(i E 1,..., ,)) a probability amplitude and E. af = 1. In practice, w can be the distributive representation or the one-hot representation for the word w. A sentence can be represented by tensor product @ Jamong words in sentence. For the representation of sentence S with n words, can be written as follows,
8 = wi x X Wn.
where (w)... w are the word which is defined in E.q. D
In order to establish the correlation between two sentences (V2 with a number (saying a) of words in the question and VA with b words in the answer), we can have the following equation. The state of the two sentences system can be defined as:
m m
(3)
j=1
'.gg 61 (1,0)", e2 = T) 61 @ 62 10,
2

---

Under review as a conference paper at ICLR 2020
where the state vector & is composite system between two sentences (i.e., Vq and Va), o is high-dimensional (m*, m the dimension of word vector) base vector for the sentence Vq. oi also high-dingg digensiinni (m²) base vector for the sentence VA. Especially, Tig the element for the high-dimensional (m² by m²) matching matrix T.
3 A FEASIBLE METHOD FOR QUANTIFYING ENTANGLEMENT ENTROPY
In this section, firstly, we use entanglement entropy to measure the quantum entanglement describing the correlation between two subsystems (c.g..a sentence pair). Secondly, we prove the equivalence between the quantum entanglement and the attention matrix under certain conditions. Finally, based on the attention matrix, the entanglement entropy quantitatively calculated, which facilitates us to design the neural network architecture through the overall inductive bias.
3.1 ENTANGLEMENT ENTROPY IN TEXT MATCHING
Entanglement entropy is used to represent correlations between two subsystems deep neural net- work (Levine et al. 2018 2019). Specifically, it requires a matrix to model all the quantum entan- glement of the two subsystems, and then the matrix is decomposed to obtain the singular values, which often correspond to the important information hidden in the matrix, and the importance is positively correlated with the singular value size. Finally, based on the singular values, the entangle- ment entropy representing the degree of entanglement between the two subsystems In the matching task, we use quantum many-body wave functions (Zhang et al. (2018b) as the basic language representation to get the entanglement entropy between sentence pairs.
be obtained.
In matching tasks (e.g., Q&& task), a Ve- - [ ..... w VA- {wii.... wA²) J. Vo E Hº with dimension m and VA E H with dimension m², respectively. m is the dimension of word vector. The composite system ths of Vo and VA, which can be written (Levine al. 2018 Cohen & Shashua 2016):
pair can be considered as two subsystems,
m m
sbs
(4)
where $1 E HP) and Φ5 HA) are basis vectors of Vo and VA, respectively. The dimensions of basis vectors about Vo and VA are m² and m², respectively. All the entries Tú, shows the cor- relation between Φ; and Φ,. When = 1, j is taken from to m², Φf represents the basis vector of i = 1 in the tensor space Hº (each base vector represents a certain combined semantics of all Q
mm" )mm, represents all base vectors in the tensor space H2, Tı.j , represents the correspond- words), {Øf 13=1
ing probability amplitude of the base vector of of the Vo subsystem combined with the base vector df of the VA subsystem. The matching matrix T can obtain all probability amplitude distributions of from 1 to m in parallel.
The matching matrix T contains all the probability amplitude distributions between the two sub- system basis vectors. Singular value decomposition (SVD) (Schollwocek & White 2013) on the matching matrix T can be written:
T Σ X.Pq. T
(5)
where sqq, and PA, are r vectors in new bases for Hº and H.A, respectively. A1 ... A are the singular values, which often correspond to the important information hidden in the matrix, and the importance is positively correlated with the singular value size, the size of the singular value represents the weight of the current basis vector of the two subsystems. A is also understood as probability amplitude, Σ, X² A represents the weight of principal semantic bases between Vo and VA. and reflects the weight of effective semantics to some extent (since it is larger than zero). Entanglement entropy (Yo 2013) computed as follows on the matching matrix T.
0
S
A A, Im|Aa 2
(6)
3

---

Under review as a conference paper at ICLR 2020
where S (0, (r), Smax = In (r), ris the number of non-zero singular values, and the number of r is called Schmidt number (Shi et al. 2006 Levine et al. 2018). The entanglement entropy describes the correlation between two systems, and the Schmidt number indicates the upper bound of the network expression ability and can assert the role of channel numbers of each layer in the overall inductive bias (Levine et al. 2018 Due to this dimensional catastrophe [ it is infeasible to calculate entanglement entropy and Schmidt number. This leads us to practically apply inductive bias is limited in the experiment. However, in matching tasks, an attention matrix often is used to describe correlation between two systems. Based on the above ideas, we propose claim to calculate the entanglement entropy.
Claim 1. Assume the compound relation of word vectors in Vq for Va) are considered in the case of 1-rrder, and it just is considered for the compound relation between each word vector in Vq and each word vector in Va. An attention matrix can be received by computing the trace of block matrices of the matching matrix T.
Proof. Assume the compound relation of word vectors in Vo(or VA) are not consider (e.g.. 1-order), the dimensions of Hº and H.A in E.q. 4 are x (am) and b (bm), respectively. E.q. can be rewritten as follow:
am
bm
T
1=
where Φ is the base vector from group of complete base vector space si..... ..... the base vector from a group of complete base vector space Φ ,....., Dan. The T is matching matrix of am x bm. Divide the matching matrix T into a x b block matrices (P., E Rmxm) )i E
22 am and of
la [b]), which can be written P.j wi Q wa Each element E,y of the attention matrix E IS equal to the inner product two word vectors which are from the sets Vo and VA, which is
E.j o w (i [a]; E [6]). There is mathematical connection between P. and Eij. which
can be written as:
E, trace Psj)
(8)
The relation between the matching matrix T and the attention matrix E has been shown as the Fig
T
P
E
Til
E
E, tr(P))
:
Pa
E
E
a.b
Tol
T
7P
7
Figure 1: The relation between the attention matrix E and the matching matrix T: Pij is the block matrix from the matching matrix T. Eig is equal to the trace of matrix P.j. T (k., [m], [a]..j E [6]) is the entry of block matrix P.j. the trace represent the summation of diagonal elements of matrix P.j.
Dimenssiona catastrophe is mainly reflected in two aspects. First, in Eq the matrix T(T Rm mm* has dimensional catastrophes. Second, even the SVD is decomposed on the matrix its number of singular values still has dimensional catastrophe (i.c., number of singular values S min(m", m ")).
[a] represents set (1, ..
4

---

Under review as a conference paper at ICLR 2020
Thus, the attention matrix E instead of the matching matrix T to quantify the entanglement entropy. In our Conjecture 1, we provide a generalized proof of the mathematical relationship be- tween the attention matrix and the entanglement properties. In the experiments, distance about two vectors to replace the inner product actually. the next subsection, we will describe our Algorithm.
use the cosine
3.2 NETWORK DESIGN BASED ON ENTANGLEMENT ENTROPY
Recall that the sentence pair contained query sentence and answer sentence be represented through sets, which are Vq and VA, respectively. where Vo (wf,...,,??), u VA
(wf'.... ..... w and b
Based on the Claim 1, the attention matrix between of Vi and VA can be computed by using cosine similarity in experiment,
w w
Ei
(9)
@ x |0na1
the attention matrix E Raxb can be decomposed by SVD.
After that, the entanglement entropy can is calculated through E.q. 6 on attention matrix E. while the upper bound of entanglement entropy Smaz obtained based on the number of singular values. The normalized value of the entanglement entropy is used in practice, it can be written as follows.
= (Smaz - SS/SSmmm
(10)
We calculate all data sample differences D, and then find the median D. of the difference D. The samples with D greater than Dm are defined as the group with the short-range correlations, while the samples less than Dm defined as the group with long-range correlations. Based the nection between deep learning and quantum entanglement (Levine et al. 2018), i.c., deep learning deseribes the correlation required for deep learning tasks, can adaptively design convolutional network by the correlation (e.g., long/short-range correlation). For the more complex the input (i.e., the long-range correlation). more kernels should be assigned in the relatively-deeper layers, while more kernels should be assigned in the relatively-shalloerr layers for short-range correlations. In our opinion, quantitatively calculating entanglement entropy by attention matrix can not only help us adaptively design the network architecture, but also provides a new perspective for us to under- stand the attention matrix in matching tasks.
EXPERIMENTS
Question Answering aims to rank the answers from a candidate answer pool for a given question. The ranking is usually based matching part between these two representations.
parts, namely, representation part for single sentence and
Datasets We validated our model through the TREC-QA (Wang al. 2007) and YAHOO-QA (Tay et al. 2017) datasets in the matching task. TREC-QA dataset contains two training sets, namely TRAIN and TRAIN-ALL. We use TRAIN-ALL, which is larger and contains noisier question- answering pairs, in order to verify the robustness of the proposed model. YAHOO-QA dataset is collected from yahoo answers for community- longer than those in TREC-QA. As introduced in this paper (Tay et al. (017)) we select the QA pairs containing questions in which the token number of its corresponding answers is more than 5 and less than 59 after removing non-alphanumberic characters. For each question, construct negative samples by ranking the top 4 answers from the whole answer sentences according to BM25.
question answering. The answers are generally
Parameters Setting We utilize the Adam (Bengio& & LeCun 2015) optimizer with learning rate 0.001 and use the best model obtained in the dev dataset for evaluation in the test set. The batch size and L2 regularization are trained by word2vec (Burges et al. 2013) on English Wikimedia dump, in which the Out-of- Vocabulary words randomly initialized by uniform distribution in the range of (-0.25, 0.25).
set to 64 and 0.00001, respectively. The 50-dimension word embeddings
5

---

Under review as a conference paper at ICLR 2020
Table 1: Results on the statistics of dataset. a, and E, denote significant improvement (with p <
0.05) over QLM (Sordoni et al. 2013) and NNQLM-II (hang et al. 2018a), QMWF-LM (Zhang
al. 2018b), CNM (Li et al. 2019 respectively, according to Wilcoxon signed-rank test.
TREC-QA
YAHOO-QA
MAP
MRR
p@1
MAP
QLM (Sordoni 2013)
0.678
0.726
0.395
0.604
NNQLM-11 Zhang ei al.
2018a
0.75y
0.825
0.466
0.673
QMWF-LM Zhang al.
2018b)
0.752
0.814
0.575
0.745%.0
CNM [Li et al. 2019)
0.770
0.859
Our-Model (adaptive setting kernels)
0.881 a,,,,,
0.924
0.630
0.789
deep-mone-kernebb-inee for whole dataset
0.668
0.762
0.342
0.574
Balloo--morre-kerll--iiixee for whole dataset
0.665
0.759
0.346
0.557
Multi-Perspective CNN (Rao et al. 2016)
0.780
0.834
Attention Pooling Networks (dos Santos et al. Holographic Dual LSTM (Tay et al. 2017
2016
0.560
0.726
0.752
0.815
0.557
0.735
Cross Temporal Recurrent Networks Tay al.
2018
0.771
0.838
0.601
0.755
Multi Multi-Perspecce Perspective CNN QC (Bender
2018
0,836
0.863
QC + RNN + Pre-Attenion (Kamath et al.
2019
0.852
0.891
4.1
EVALUATION METRICS
TREC-QA is evaluated by MAP(mean average precision). MRR(mean reciprocal rank), while YAHOO-QA is evaluated in term of P1 (precision at one) and MRR since it targets more the first ranking item.
4.2 CONFIGURATIONS FOR OUR MODEL
In this paper, we adopt the quantum many-body wave functions (Zhang et al. 2018b) the basic representation components, to represent both question and answers sentence as two subsystems i.e. Q and respectively. Since the whole system with guestion-answering pairs is composed by these two subsystems, the entanglement entropy is adopted to quantify correlations between subsystems.
Generally speaking, there are typically two kinds of correlations in question answer task, namely long-range correlation and short-range correlation. The former refers to more complex matching with consideration of the global sentence. The latter refers to the cases that there are already over- lapping words phrases between the question sentence and the answer sentence, which can be matched locally at the lexical and syntactic level. As discussed our algorithm, the entanglement entropy is quantitative metric distinguish how much semantically entangled the question and answer are. Thus we could divide question-answering pairs into two sub-datasets, i.e. long-range sub-dataset and short-range sub-dataset. Then, ferent sub-datasets. For long-range sub-dataset, we should choose the more kernels in deep layers (namely seep-more-kenell mode). while for short-range sub-dataset, we should choose the more kernels in the shallow layers (namely worooo--oo----eeeeeeee mode). At last, we combine the evalua- tion scores of the long/short range sub-datasets as the data for the whole dataset to be evaluated.
design an adaptive setfing for kernels for dif-
We set parameter from following modes for an input layer and three-layer CNN network:
(1) Malloo-more-keenen [m,3r.2r.m] (2) deep-more-kernete (mm
m is hyper-parameter for the kernel count of the last CNN layer, which is set as dimension of word embedding for guaranteeing the same parameter scale between these two modes. The parameter count for both configurations is identical: m 3r + 3r 2r + -m = 6r 5mr. Shallow-more- kernels mode has more kernels in the lower layers while the deep-more-kernels mode has more kernels in higher layers. The code will be open-sourced in
Comparative Setup To validate the above adaptive setting for parameters, design compar- ative fixed settings without dividing the dataset. Fixed setting 1: the happ-more--eeeee parameter mode is used for the whole dataset. Fixed setting 2: the shallow-more-kent kernels mode is applied to the

---

Under review as a conference paper at ICLR 2020
TREC-QA
YAHOO-QA
MAP
0.8
0.7
066
0.8
0.5
Number of convolution karnals
Number of convolution kemels
-our-modee
deee--are--emmals
ndon So-thallowmorr--erann
oun-modal (adaptive)
(adaptive)
(Soedd
(ficed)
(fised
(fised)
TREC-QA
YAHOO-QA
0.6
0.89
0.55
0.79
0.03
0.3
16
32
96
128
16
96
128
Number convolation kemels
Number
shallow-more-keradd (fived)
our-model (adaptive)
1D-eep--orce more--all-sssss
we--halllowwmee keranls (ftxed
(ndaptive)
(fxxe)
(Ased]
Figure 2:
We compare our model (adaptive settings for kernels) to two fixed modes, deep-more- kernels for the whole dataset and shallow-more-Kee kernels for the whole dataset. For TREC-QA, model with an adaptive setting for kernels achieved significant improvements on MAP and MRR (with p < 0.01) over the fixed settings. For YAHOO-QA, can get similar results. Both the statistic tests are based on Wilcoxon signed-rank test.
whole dataset. Except for the setting for the kernel numbers and dataset paritition, Other parameters in both adaptive and fixed settings are the same, for a fair comparison.
4.3 RELEVANT METHODS FOR COMPARISON
QLM (Sordoni et al.
2013) adopts the density matrix of quantum mechanics to model the depen- dencies between words such as query words and documents.
2018a) is a end-to-end quantum-like language model for question answer- ing. It adopts convolution neural network over the multiplication between two density matrices.
NNQLM-II [Zhang et al.
QMWF-LM (Zhang et al. 2018b) is based on the quantum many-body wave function representation and further convolution neural network for approximate tensor decomposition.
CNM et al. 2019) utilizes the probability properties of the density matrix, indirectly measures the distance between two density matrices through a trainable measurement operation, and uses training data to provide flexible matching scores.
Our model is also compared series of CNN and RNN models (Rao al. 2016 Tayyar Mad- abushi et al.) 2018 Tay et al. 2017] 2018 Kamath et al. 2019). To the best our knowledge, Question Classification RNN Pre-Attention (Kamath et al. 2019) is the state of in the raw
version of TREC-QA.
4.4 EXPERIMENTAL RESULTS
Tab. reports the results on TREC-QA dataset and YAHOO-QA dataset. The first group shows a comparison of between model with four quantum-inspiree language models. For TREC-QA, our model significantly outperforms QLM by 20.3% on MAP and 19.8% on MRR, respectively. Compared with NNQLM-II and QMWF-LM, CNM, it also reflects the superiority of our model.
hhnns://eeewwhonnn/aastiittttttn Answering (State of the art)

---

Under review as a conference paper at ICLR 2020
For YAHOO-QA -A, model is significantly better than QLM 23.5% and 18.5% in P@1 and MRR, significantly better than NNQLM-II 16.4% and 11.6% in P@1 and MRR, and better than QMWF- 5.5.5% and 4.4% in P@1 and MRR. In summary, compared with three -onnntu--nnspreed language models, our model shows good experimental performance on both tasks.
In the second group, our model with adaptive setting for kernels is compared with other two fixed settings, i.e., deep-more-kernels fixed for the whole dataset and maallo--more-keneee dataset (see Sec. 4.2 for details). For TREC-QA, the results show that both MAP and MRR are at least 15% better than them. For YAHOO-QA, the results show that both P@l and MRR are at least 20% better than the comparative setup. In summary, the results reflect the effectiveness of the adaptive settings of parameters for two divided data-subsets.
for the whole
In the third group, we compare our model with number of CNN-based models and RNN-based models (Rao et al. 2016; Taayarr Madabushi et al. 2018 Tay et al. 2017: 2018). Multi- CNN (Rao et al. 2016] and Multi-Perseecivve CNN +PairwiseRank + Question classification (Tay-] yar Madabushi et al. 2018) model the QA task ranking task and proposes a pairwise ranking method that can directly utilize the existing point-by-point neural network model as a basic compo- nent, HD-LSTM (Tay et al., (017) and CTRN (Ta et al. 2018), Question Classification + RNN Pre-Attention (Kamath et al., 2019) propose an RNN-based approach to match conestion-answerinn pairs. It should be emphasized that our model is better than Question Classification RNN + Pre- Attention (Kamath et al.] 2019), which is the state of art in the raw version of TREC-QA. For TREC- QA and YAHOO-QA,
models are better than the results (Rao et al., 2016 Tayyar Madabushil
ctt al.
2018 Tay et al. 2017: 2018 Kamath et al. 2019) reported in the original paper.
4.5 DISCUSSION AND ANALYSIS
Analysis of Adaptive Settings In Fig. we investigate the difference between the adaptive set- ting in our model and two fixed settings introduced. Recall that in our model, we first identify the correlation between sentence pairs based on the attention matrix and entanglement entropy. Then, according to the correlation, the question-answering pairs could be divided into two sub-dataset, ie, long-range sub-dataset and short-range datasets. Therefore, we can adaptive set the kernels for different sub-dataset. On the other hand, the two fixed settings apply the same parameter setup (deep-more-keme) kernels or horroo---orr--eeeell for the whole dataset.
As we can see from Fig. 2 TREC-QA, in our model, with the increase of the parameter MAP, MRR and P@1 show an upward trend. For the fixed settings, i.c., deep-more seep-more-kemine kernels for the whole dataset changes,
and shallow-mon more-kernels for the whole dataset, it does not show better performance and even show a downward trend. This suggests that our adaptive kernels settings can well capture the language matching characteristies of the question-answerrin pairs, and the alternative approach to the entangle entropy as the measurement for the language correlations, work well for dividing the dataset into two subsets, reflecting different ranges of language correlations.
Inffuence of Channels in Convolution In Fig. 2 as r grows, the MRR and MAP of TREC-QA and YAHOO-QA increases. We select the optimal in range [16, 32, 64, 96, 128). For different datasets, we set same number of channels to obtain the final performance. In our model, for the TREC-QA and YAHOO-QA, when the parameter r is set 128, the experimental results as shown in Tab. 1 can be obtained.
Efficiency Analysis we utilize adaptive settings of kernels for different sub-datasets. The efficiency relies on the deep convolution neural network. In our experiment, for TREC-QA, the training epoch is set to be 100, while for YAHOO-QA, after training 30 epochs, we will obtain the results.
CONCLUSIONS AND FUTURE WORK
In this paper, we aim to extend previous studies on the theoretical and practical use of the concept of quantum entanglement in neural network. We demonstrate the connection between the entangle- ment entropy and the attention matrix in text matching tasks. This allows us to quantitatively calcu- late the entanglement entropy, which helps adaptively setting network structures and parameters, and achieve effective performance on two typical QA tasks. In future work, we will investigate the entanglement entropy under high-order conditions and in other networks, e.g., recurrent networks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Emily M. Bender, Leon Derczynski, Pierre Isabelle, and III (eds.). Proceedings of the 27th Inter- national Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, 2018. Association for Computational Linguistics. ISBN 978-1-948800 1778---94887-500--
Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Christopher C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.). Ad- vances in Neural Information Processing Systems 26: 27th Annual Conference on Neural In- formation Processing Systems 2013. Proceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, 2013.
Z. Cai and J. Liu. Approximating quantum many-body wave functions using artificial neural net- works. Physical Review B, 97(3):035116 January 2018.
Giuseppe Carleo and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 355(6225):602-666, 2017.
Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry. CoRR, abs/1605.06777 06743,, 2016.
Cícero Nogueira dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. Attentive pooling networks. CoRR, hls/1602.03609 2016.
Sanjay Kamath, Brigitte Grau, and Yue Ma. Predicting and Integrating Expected Answer Types into Simple Recurrent Neural Network Model for Answer Sentence Selection. In 20th International Conference on Computational Linguistics and Intelligent Text Processing, La Rochelle, France, 2019.
Yoav Levine, Or Sharir, and Amnon Shashua. Benefits of depth for long-term memory of recurrent networks. CoRR, abs/1710.094. 2017.
Yoav Levine, David Yakira, Nadav Cohen, and Amnon Shashua. Deep learning and quantum entan- glement: Fundamental connections with implications to network design. 2018.
Yoav Levine, Or Sharir, Nadav Cohen, and Amnon Shashua. Quantum entanglement in deep learn- ing architectures. Phys. Rev. Lett., 122:065301, Feb 2019.
Qiuchi Li, Benyou Wang, and Massimo Melueci. CNM: an interpretable complex-valued network for matching. In Proceedings of the 2019 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4139-4148, 2019.
Jinfeng Rao, Hua He, and Jimmy J. Lin. Noise-contrastive estimation for answer selection with deep neural networks. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA. October 24-28, 2016, pp. 1913- 1916, 2016.
UIrich Schollwoeck and Steven R White. 16 dmrg. Ground states, time evolution and spectral functions. 2013.
Y. Shi, L. M. Duan, and G. Vidal. Classical simulation of quantum many-body systems with a tree tensor network. Phys. Rev. A74(2):022300, 2006.
Alessandro Sordoni, Jian-Yun Yun Nie, and Yoshua Bengio. Modeling term dependencies with quantum language models for ir. In Proc. of SIGIR, pp. 653-662, ACM, 2013.
Yi Tay, Minh C Phan, Luu Anh Tuan, and Siu Cheung Hui. Learning to rank question answer pairs with holographic dual Istm architecture. Proc. of SIGIR, pp. 695-704. ACM, 2017.
Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. Cross temporal recurrent networks for ranking question answer pairs. In Second AAAI Conference Artificial Intelligence, 2018.

---

Under review as a conference paper at ICLR 2020
Harish Tayyar Madabushi, Mark Lee, and John Barnden. Integrating question classification and deep learning for improved answer selection. In Proceedings of the 27th International Confer- on Computational Linguistics, pp. 3283-3294, Santa Fe, New Mexico, USA, August 2018.
Association for Computational Linguistics.
Mengqiu Wang, Noah A Smith, and Teruko Mitamura. What is the jeopardy model? quasi- synchronous grammar for qa. In Proc. of EMNLLP-CONLA. volume 7, pp. 22-32, 2007.
Yoshiko Kanadaen Yo. Entanglement entropy and schmidt number as measures of delocalization of a clusters in one-dimensional nuclear systems. Progress of Theoretical and Experimental Physics, 2015(4), 2015.
Peng Zhang, Jiabin Niu, Zhan Su, Benyou Wang, Liqun Ma, and Dawei Song. End-to-end quantum- like language models with application to question answering. In Proc. of AAAI, 2018a.
Peng Zhang, Zhan Su, Lipeng Zhang, Benyou Wang, and Dawei Song. A quantum many-body wave function inspired language modeling approach. In Proceedings of the 27th ACM International Conference Information and Knowledge Management, pp. 1303-1312.. ACM, 2018b.
A ATTENTION MATRIX AND ENTANGLEMENT ENTROPY
Given a sentence pair with word embedding {w, E [i E [n]). which can be represented using VA {wf, "w.,...,u ".,....,),,..
two sets, ie., Vq= {wi ..... ) ........ Conjecture 1. Assume the compound relation of word vectors in VA are considered in the case of n-order, and just is considered for the compound relation between each word vector in VQ and words vectors in VA. The atlention matrix can be received by computing the trace of block matrices of matrix T.
Proof. Firstly, the mathematical relationship between the attention matrix and the quantum entan- glement matrix is proved under the 1-order condition in 3.1
Assume the compound relation of word vectors in VA are considered in the 2-order, the dimension of H. is x (am) and H is Cf xm?, i... C3 (O} is Combination number") E.q. can be rewritten as follow:
Cf m
sbs Σ
T
2 0
(11)
lj=1 are base vectors for 7 and _ The T is martix of am x Cf Divide the T into x C block matrices (Pa E pmxm" ]i E [o]:jj E [C)]), which can be written P.j t s n. und (wy E TRT" represents the compound semanties of any two-word vectors in VA. Matrix P.y can be regarded as a 3-order tensor Pi (Ptttts E (mmxx")) each element Eij E [a]: (CH) of the attention matrix E is equal to the sum of the 3-order tensor Pttta diagonal elements, i.e., E.j- =< wh wiz wa > (t) E [a], l2 o t3 and 12, t3 E [b]). there is mathematical connection between Pu and Eij. which can be written as:
aam where ( Timl and ( Cmm²
Eij tracee
(12)
Thus, we use the attention matrix E instead of T to quantify the entanglement entropy.
Secondly, assume the compound relation of word vectors in Vo VA) considered in the k-order (k < b), the dimension of 2 is axm (am) and H is Cj xmt (*)m*). E.q. can be rewritten as follow:
C
Ti 00
(13)
6mm represents the number of combinations of elements taken from different elements(m n)
10

---

Under review as a conference paper at ICLR 2020
T
E
Pa
P
E 11
E
E =
:
Pa P
E
E
c=
C
Figure 3: Assuming the compound relation of word vectors in Vo with a words (or Va with b words) considered the 2-order which represents c (c C% compound semantics of any two-word vectors in VA. The relation between the attention matrix E and the matrix T: P.j which can be regarded as 3-order tensor is the block matrix from the matrix T. Eij is equal the trace of matrix P.j (i [a]; E [c]).
Cf ( jj=1 base vectors for HA and H The T is matrix of am Cj Clm² Divide the T into C block matrices (Pi E m }i [a]:j E [CH1), which can be written as P.j- = w X wj wj (wy² E R= represents the compound semantics of any & word vectors in VA. Matrix P.j can be regarded as a k-order tensor Ptita..xx (Prits..ta E <Imxmx-xx)) each element Esj (i E [a]i E (Ct]) of the attention matrix E is equal to the trace the k-order tensor, i.e. Egg =< wt, w w ... w > (11 E [a), + ta + ... + tu and 12, t3.... tx [b]), which can be written as:
where { aam and {
Eij trace (P.))
(14)
Thus, we use the attention matrix E instead of T to quantify the entanglement entropy. Thirdly, assume the compound relation of word vectors Vo VA) are considered in the (k: orrde
(I: < b), the dimension of Hº is ax m (am) and H is Cg h+1 (C&++ h+1. E.q. 4 can be rewritten as follow:
am C. Chitl m 611
Σ
T2 001
(15)
j=1
where { 1a
Cf m Jimi and (Φ 1==1
are base vectors for H and Hº The T is a matrix of am x
Cht m k+1 Divide the T into a x c++1 block matrices (Pay E xmmxm*)) Hi [a]; E [C which can be written P.j w w/ w (wf R represents the compound semantics of any (k++) 1) (Prttanatuee) E R=xmxx-xxX) each element Eij (i E [a]: E [CR+1] of the attention matrix E equal to the trace of the (k: + 1)-order tensor, i.e., Eij =< w w w ...... (.... kk+1) (t, E [a], 12 + to + ((x+1) and t2, ts. ... tts....((+1)) [b]), which can be written as:
word vectors in VA- Matrix P.j can be regarded as (k+ 1)--derr tensor Prtta..taaa
Ei ttrace(Ps)
(16)
Thus, use the attention matrix E instead of T to quantify the entanglement entropy.
In summary, the equivalence relationship between the attention matrix E and the entanglement prop- erties matrix 7 is satisfied under the 1-order condition, the 2-order condition, the k-order condition, and the (k+l)-order condition, which proves that the relationship is established. That is, attention and entanglement properties equivalent under specific premise.
11


Reviews:
[{"review_id": "ryeIgj5pFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with the understanding of deep learning under a physical point of view, related to entanglement entropy.\nAs far as i understand the paper explains how the computation of the entanglement entropy may be performed and how this measure may be used to design the neural network architecture. \n\nI was pretty much interested by the paper but I was unfortunately not able to follow the « theoretical » part of the paper, which in my opinion is not well introduced. The paper cites papers from physical letters and physical reviews without introducing all necessary background. From this point of view the paper is not self content enough for the audience of the conference.   \nI don’t know what is the quantum many-body problem, never heard of StartEnd separation rank… mentioned in the first paragraph of the paper.\n\nIn addition to the fundamental contribution that i could not summarize well, the paper includes an experimental validation of the contribution on the question answering problem on benchmark datasets, showing very relevant results outperforming the baselines, which look like state of the art methods in the field. \n\nNot sure to fully understand the experimental setting though. In particular i understand that the dataset are divided in isolated parts on which different models are learned. And averaged results are reported. But is it a fair comparison with the baselines ?   \n\nAt the end the paper looks quite interesting and promising but in its current shape it doesn’t seem to me fully accessible to ICLR audience. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJlrd6mUcS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work extends the Quantum Many-body Wave Function inspired language model (QMWF-LM) of Zhang et al by proposing some quantum entanglement entropy computation to separate the data into long range correlation and short range correlation. The authors report improved results on the TREC-QA and YAHOO-QA datasets.\n\nI am not an expert on Quantum physics, hence I am unable to judge the merits of the quantum entanglement approach proposed in the paper. However, the work that this paper builds on - the \"QMWF-LM of Zhang et al\" - doesn't seem to have been vetted by proper peer review in either an ML conference or journal. I am also skeptical of the quantum terminology introduced in the paper and the experiments are reported on only two QA datasets - TREC and YAHOO which aren't super standard. If Quantum inspired language models are the next big progress in language modeling, I would like to see more experiments on some language modeling datasets such as LM1B (Chelba et al), Wikitext-2/103 (Merity et al) etc. Besides, this approach should also work for SQuAD, GLUE, SuperGLUE and all the other established NLP benchmarks that benefit from improved language modeling capabilities.\n\nI am also skeptical of the progress in TREC-QA, the state-of-the-art claimed by the authors is Kamath et al which uses an RNN + pre-attention. A stronger and more modern baseline would be something like BERT (Devlin et al.) or any of the subsequent improvements to it.\n\nThis paper would benefit from a clearer exposition minus the quantum mechanics jargon and from experiments on the above stated benchmarks to be more convincing.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJe8f0JRKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a method for leveraging entanglement entropy for understanding attention matrices is proposed. Specifically, the paper aims at solving two problems: 1) to study the theoretical analysis of entanglement entropy for the matching of two objects (question-answering pairs), and (2) to qualitatively calculate the matching matrix. The introduced approach is based on fundamental connections between the entanglement entropy and the attention matrix. The main goal of the paper is to show that a low-dimensional attention matrix can be derived from a high-dimensional matching matrix. Results are shown for a text matching task on two datasets (TREC-QA, YAHOO-QA). \n\nMy main concerns with this paper are that the approach is not well described and that the proposed contribution appears quite narrow. Overall it is not clear what the actual contribution of this work really is. Several sections of the paper appear convoluted and could be more concise; the text is often written unnecessarily complicated which makes it hard to follow and to comprehend details. While the experiments seem sound the overall improvement of 2.9% compared to SOTA work appear rather shallow. Therefore, in its current state I cannot recommend accepting this paper. \n\nDetailed comments: \n\n- The abstract of the paper is very cryptic and the motivation for the proposed approach is not sufficiently described. \n- The paper is not well-written and difficult to follow. Several sentences and sections are left too unclear. E.g. in the introduction sentences like \"... but such an indicator only reflects the intricate correlation structures of each single input object (e.g., an image or a text)\" or \"This is due to the fact that the tensor product occurs in the quantum many-body function for representing the image and text \" are confusing as they come without much context. \n- It is not clear what is meant by \"matching problem\".\n- A few sentences in the introduction are exact copies of the abstract, which makes the text appear redundant. It would help to clearly state what the contribution of this work is, instead of repeating sentences. \n- It is not clear what is meant by \"relatively-deeper\" and \"relatively-shallower\" layers.\n- The text shows several spelling and grammar issues (such as \"for the more complex the inputs\"). \n- Several sentences don't make sense and are difficult to read (e.g. \"Since our work is mainly for the text matching task of a sentence pair, we briefly introduce a recent Quantum Many-body Wave Function inspired Language Modeling\").\n- The notation and equations in Section 2 are mostly common knowledge and could be moved to the appendix. \n- In Section 3.1. what is meant by \"subsystem in deep neural networks\"? Later in the text in becomes more clear. So it would be helpful to rearrange the text. \n- \"probability amplitude distributions \" is not clear. \n- Sentences like \"which often correspond to the important information hidden in the matrix\" should be accompanied with a reference. \n- Section 3.2 is titled \"Network Design Based on Entanglement Entropy\", but the section does not actually describe a network architecture, but instead just describes how to obtain the attention matrix and the sample differences. \n- Section 4.1 (first paragraph): it would help to discuss the connection of many-body wave functions to represent questions and answer sentences as two subsystems more clearly and earlier in the text. \n- Section 4.2 (second paragraph) appears quite repetitive. Many sentences have been used in previous sections of the text. \n- The results and discussions shown in Sections 4.4 and 4.5 are interesting and seem sound. However, it is not clear what exactly are the \"adaptive settings for kernels\" (e.g. in Figure 2) \n- Limitations of the approach are not sufficiently discussed. \n- It is not clear what is meant by \"we will investigate the entanglement entropy under high-order conditions\" in the conclusion.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper advocates for the application of entanglement entropy from quantum physics to understand and improve the inductive bias of neural network architectures for question answering tasks. All reviewers found the current presentation of the method difficult to understand, and as a result it is difficult to determine what exactly the contribution of this work is. One suggestion for improving the manuscript is to minimize the references to quantum entanglement (where currently is it asserted without justification that entanglement entropy is a relevant concept for modeling question-answering tasks). Instead, presenting the method as applications of tensor decompositions for parameterizing neural network architectures would make the work more accessible to a machine learning audience, and help clarify the contribution with respect to related works [1]. 

1. http://papers.nips.cc/paper/8495-a-tensorized-transformer-for-language-modeling.pdf
Decision: Reject