Jump Self-attention: Capturing High-order Statistics in Transformers
Authors: Haoyi Zhou
Siyang Xiao
Shanghang Zhang
Jieqi Peng
Shuai Zhang
Jianxin Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Network
Transformer
Self-attention

Abstract: The recent success of Transformer has benefited many real-world applications, with its capability of building long dependency through pairwise dot-products. However, the strong assumption that elements are directly attentive to each other limits the performance of tasks with high-order dependencies such as natural language understanding and Image captioning. To solve such problems, we are the first to define the Jump Self-attention (JAT) to build Transformers. Inspired by the pieces moving of English Draughts, we introduce the spectral convolutional technique to calculate JAT on the dot-product feature map. This technique allows JAT's propagation in each self-attention head and is interchangeable with the canonical self-attention. We further develop the higher-order variants under the multi-hop assumption to increase the generality. Moreover, the proposed architecture is compatible with the pre-trained models. With extensive experiments, we empirically show that our methods significantly increase the performance on ten different tasks.

Jump Self-attention: Capturing High-order Statisties in Transformers
Haoyi Zhou BDBC Beihang University Beijing, China 100191 haoy1 @buaa. edu. en
Siyang Xiao BDBC Beihang University Beijing, China 100191 xiaosyEact, buaa edu en
Shanghang Zhang
School of Computer Science
Peking University Beijing, China 100871 shanghang@pku. edu cn
Jieqi Peng BDBC Beihang University Beijing, China 100191 pengjqlact buaa. edu, en
Shuai Zhang BDBC
Jianxin Li BDB Beihang University Beijing, China 100191 1 1ijx@buaa edu. cn
Beihang University
Beijing, China 100191 zhangs@act act buaa. edu, on
Abstract
The recent success Transformer has benefited many real-world applications, with its capability of building long dependency through pairwise dot-products. However, the strong assumption that elements are directly attentive to each other limits the performance of tasks with high-order dependencies such natural language under- standing and Image captioning. To solve such problems, we are the first to define the Jump Self-attention (JAT) to build Transformers. Inspired by the pieces moving of English Draughts, we introduce the spectral convolutional technique to calculate JAT the dot-product feature map. This technique allows JAT's propagation in each self-attention head and is interchangeable with the canonical self attentttenion. We further develop the higher-order variants under the multi-hop assumption increase the generality. Moreover, the proposed architecture compatible with the pre-trained models. With extensive experiments, empirically show that methods significantly increase the performance on ten different tasks.
Introduction
Transformers have made great success in many domains, such as NLP, Vision, Time-series, and Speech. However, the giant model GPT-3 [2] and switch Transformer 77 with their massive layers and over-parameterized architectures, have not further achieved significant performance improvements since the BERT model beat the Human Performance in GLUE [21] and SQuAD [12]. Recently two visualization works [19 reveal that the self-attention mechanism mainly learns superficial connections among inputs. The principle of direct accessing among inputs in the dot- product self-atte self-attention lies the core of above models, which solely relies their mutual similarity in the vector space. However, as "every coin has two sides", while the dot-product simple and effective in capturing long-range dependencies, it has limited capability in learning high-order dependencies.
To solve this problem, we are inspired by the pieces moving English Draughts, a checker game spread to Great Britain in about 1100. It allows two moving rules: one is the simple move from the
JJinxin Li is the corresponding author.
BDBC is the abbreviation for the Beijing Advanced Innovation Center for Big Data and Brain Computing.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Dot-product attention with direct connections.
(b) High-order attention with "jump" connections.
(a) The dot-pre dot-product self-attention: Left side: The direct connections (the blue lines with arrows) are established between the input A (the red node) and other inputs via the dot-product self- attention. Right side: The nodes with different shades of blue represent the corresponding attention of the feature map. (b) The high-order self-antention: Left side: We build "jump" connections from input A (the red node) to previous unattended nodes at the corner, which is conditioned on the direct connections on "stepping stones" (the green circled nodes). Thus the dot-product feature map is enhanced with high-order connections as shown on the right side.
Figure I:
current position to an adjacent position; and the other is the capture move jumping over the "stepping stones" (opponent's s pieces) from front back. Multiple jumps are possible and forced in this game. As shown in Fig. [ we can employ the high-order self-attention connections, like the capture moves in English Draughts, enhance the feature map of dot-product self-attention. It helps align high-order global dependencies in Transformer models. As a more concrete example in natural language understanding, we are given two sentences "Book costs more than pencil." and "Book costs less than computer". "Computer" never attends to "pencil" until they were both compared with "Book", thus "Book" is the "stepping stones" creating high-order attention between "computer" and "pencil" in Fig(Tb). More specifically, they forms soft triadic closure. We will discuss this example in detail in Section 3.1 and clarify the definition of Jump Self-attention.
In this work, we find that the high-order self-attention exists in the dot-product computation but it is weak contribute much to the attention layer's output. Only few heads in Transformers can capture such high-order dependencies, which is shown in Fig. (2) with further discussions. Instead of finding such "miracle" heads. we reformulate them to enhance the high-order dependencies with the feature propagation technique in graph learning. Meanwhile, we build an efficient variant in computing sensitive situations based on sparsity measurement 25 and build higher-order variants in comprehensive situations using MixHop [1]
To the best of our knowledge, we are the first to define the Jump Self-attention (JAT) Transformer models, where we formulate the high-order statistics calculation into a graph propagation problem. It can be used interchangeably with the canonical self-attention [18]. The JAT-enhanced Transformer allows for fine-tuning on the popular pre-trained models and achieves superior performance on various tasks, such as Natural Language Understanding and Question Answering. Our main contributions can be summarized as:
We discover the existence of JAT in the Transformers' dot-product computation and derive its formal definition to facilitate further analysis.
We propose JAT through the jump aggregation and high-order dependencies' propagation without additional network parameters, and the efficient variant alleviates the extra computation cost.
We propose the higher-order variants of JAT and masking strategy allow its interchangeable usage with the canonical self-attention on the pre-trained models.
We conduct experiments on fifteen different tasks to examine the effectiveness of the proposed methods. Results demonstrate the success of JAT in achieving state-of taiee-of-he--- t--thh--rt performance.
2 Preliminary
The self-attention was introduced by Transformer [8 model for the encoder-decoder architecture in NLP to build long-range dependencies from a variable-length source sentence. The alignment ability of self-attention to learn to acquire direct access between important items has made it an
efficient component in many applications [5] 1 25 23
For an input E Lxddd we define the
single-headed self-attention in the following form:
QK
Attention Q, K, V) softmax
(I)
vd

---

bood
book
25
Computer-Pencil Pencil
attention connections
1000
Computer Pencil Book Pancil
Book- Computer
80
Book Computer
20
connections
connections
than
then
penci
pendl
sttenttoo
[SSP]
[BEP)
10
40
Etase
lack
20
105
than
Hen
101:12
101112
The layer of self-attention
The layer self-attention
(a) Attn [layer7, head5). (b) Significant connections (15.2%).
(c) Major connections (70.5%).
Figure 2: A case study of sentence "Book cost more than pencil. Book cost less than computer." on the pre-trained BERT model [5]. (a) Attention Visualization: We specifically select the third layer and draw the attention score of two words "book" and one word "computer" the fifth head in the multi-head self-attention. It is an extraordinary example to show that the word "computer" in the second sentence has established moderate connection with the word "pencil" in the first sentence. The visualization tool is available online" (b) Statistic of self-attention for significant connections across all heads: We first calculate the mean value and standard deviation over all the pairwise self-attention scores. And we select the 15.2% of the self-attention connections among the three words "book", "pencil", and "computer" that their scores are greater than the threshold (mean 0.5 std). The previous example in (a) is the miracle attention connection at the third layer, and it works by acquiring more connections than other layers. (c) Statistic of self-attention for major connections across all heads: We lower the threshold to (mean 0.5 std) and it covers 70.5% self-attention connections. shows that the dot-product self-attention attempts to build the ccomputer-pencil connections, but they are a few weak ones and contribute little to layer output.
where the layer output is the linear combination of values weighted by the attention scores between queries Q and keys K. Note that the queries, keys and values are projected by different linear transformations { { o: WK, Wv} respectively. In practice, multiple attention heads are used to learn robust patterns [3 [9] among different transformations. It divides the inputs into H groups and computes the single-head attention on each group separately. We concatenate the feature map of all groups as the attention layer output.
3 Methods
In this section, we first introduce the necessity of acquiring the high-order connections in the Transformer model. Then, we describe the proposed Jump Self-attention.
3.1 What is the High-order Statistics?
The BERT [ model and its pre- rreitrinned successors utilize the self-attention, like GPT-3 2 and switch Transformer M7) and they achieve distinet performance improvements on various NLP tasks. However, what the self-attention mechanism has learned remains to be a challenging problem for architecture optimization Transformer. Recent visualization techniques on self-attention mechanism have shed light on the further explanation for this problem. In the BERT model, Vig [19 pointed out that different heads of the same layer highlight coarse positional patterns, such as 'CLSS' or 'SEP' tokens, more frequently than lexical patterns, even though they are trained with separate random projections. Clark et al. (3) claimed that BERT's attention heads often exhibit similar behaviors within the same layer, such as attending to delimiter and positional tokens and broadly attending the entire sentence, whereas a few deviating heads hold an eye on linguistic pairs. Due to the space limit, more comprehensive related work will be provided in the Appendix A. This paper investigates whether is worth enhancing lexical patterns in self-attention.
To motivate our approach, we perform an individual case study on investigating the lexical patterns of self-attention heads. Based on the pre-trained model on BERT ( we consider logical reasoning questions on two continuous sentences "Book cost more than pencil" and "Book cost less than computer"" seeks to find the price between entities that have not been compared, like "pencil"
'The tool can be acquired atl.tpps//kKau...mmm/eeeerr//ees

---

and "computer". Note that the articles are omitted to avoid interference. In Fig. 22), we present the visualization of three words' self-attention scores at the third layer and fifth head. The darker line color denotes higher self-attention score, namely stronger connections. The word "book" shows strong attention connections (deeper blue lines) to the positional token 'SHP' the first and second sentences, meanwhile, they also attend to the entity words "pencil" and "computer" (lighter blue lines). While the attention scores between the word "computer" and token 'SEP' are striking, we note that the word "computer" builds a mild attention connection to the word "pencil" crossing two sentences. From the entity extraction perspective, the mutual attention connections among three words "book", "pencil" and "computer" seem trivial.
However, the concrete finding in Fig.(p) undermines this emotional thinking and it reveals the potential value of attention connections between "computer" and "pencil". We first calculate the mean value and standard deviation over all the pairwise self-attention scores. And we select the attention connections that their scores are greater than the threshold (mean + 0.5 std). We only draw the local connections among the three words and it covers 15.2% of the total as the significant local connections. The stronger "Computer-Penciis self-attention computation that happens at the third layer, which distinguishes it from the trivial attention connections. As we discussed above, there are many connections from three words to the delimiter 'CLS' and positional tokens 'SEP' but they did not form the "Computer-Pe Pencill connection. Furthermore, in Fig. (2p). there are many connections and "Book- Coooo--Cmmpter" connections that exist in other layers but they all fail to capture the "Computer-Penci""" connection. The fifth head in the third layer is the miracle head that can establish the strong linguistic connection between "computer" and "Pencil" and it forms lexical self-attention patterns across different sentences. We call this exact example of the High-order Statistics, and it formulated on input pairs which satisfy the following definition:
attention connection, it in Fig 2p), is the rare
"Book-pencil and "Book-Compute
Pencil"
Definition 3.1. (8. p. W Q. W-)-imiilr For vector inputs x,, x, and X3, there exists a single head self-attention with query transformations We and key transformations WK and the self- attention computation forms the feature map score as Sij- = (x)))) Wx) such that, under the ö-significant threshold p.
is Sik 2 p and Sjz 2 p. then Pr(S.j 2 p] > 8
otherwise, then Pr(S.j 2 pl S 1 8
Thus the inputs x; and x, called (.,p, Wo.
w.r.t the input Xx:-
In Fig. 2p), we relax the restriction on significant connections and select the major connections covering 70.5% local connections among the three words. As natural result of decreasing the significance 5 and threshold p in Definition 3.1. it emerges more high-order self-attentions, i.e., connections between "Computer" and "Pencil". This implies that weak high-order self self-attentions exist in the canonical self-attention but these connections are too delicate to contribute to self- attention's 's layer out.
3.2 Calculate the Jump Self-attention
The Jump Self-attention (JAT) is based on the vanilla Transformer architecture to be consistent with the latest improvement on it. Inspired by the pieces moving in Fig. we build a transaction graph on the self-attention feature map. Then JAT utilizes the GCN operator allow "jump" connections to traverse through different queries and key pairs, and the weak connections under the similar condition enhanced to help enforces the desired triadic closure property.
Let the attention score of single-headed self-attention in Eq. I be S QKT, T we notice that the softmax(-) :) function on S/ V is row-wise, which makes the self-attention layer output the correct re- representation of value V. To understand the underlying meaning of high-order statistics in attention computation, we can take a column view of S in Fig. (3)). Different from the row view where each row reflects one query's 's scores on all keys, in the column view, each key column represents similarity measurement of all queries against itself.
Consider an intuitive example from the previous case study, we draw the attention feature map with five queries and keys (3). We firstly present the dot-product self-attention scores on the left side. The self-attention scores S31 = Q2(K) of the first key, as we can see the crosshatch under the column view (the red shade), which indicates that inputs x3 and x4 attend to the input x; from the K, 's observation. The connection between X3
and S41 (K(),) highlight in the column vector

---

Book
Querles
Queries
Qyl Pencil
Qui Pencil
Qui Compute"
Jmm
Out Computer
Keys row view
Keys
Kut Book
Ki Bonk
columo
Ka Computer
(a) The different views.
(b) The overview of the high-order self-attention.
Figure 3: (a) Two different views on self-attention scores: We present the row view of self-attention the top and the column view at the bottom. The solid blue line denotes the second query's self-attention distribution over all the keys, and the single apex indicates focused attention. The solid green line represents the self-attention distribution between all queries and one key, and the multi-peak reveals three stronger attention connections for that key. (b) An example of finding the "jump" connection: We utilize the "Pencil" and "Computer" attention "Book" enhance closer similarity between them, namely, from the crosshatch to the red marked squares.
and X4 are weak (the blank square). Based on the high-order statisties, we have a high probability ó that the inputs X3 "Pencil" and X4 "Computer" becomes (6. p, Wq. W W-)-similar w.r.t the input X1 "Book". But the dot-product self-attention can not capture this kind of similarity for and x could modify the linear transformation Wo and W K such that the red marked square attains larger attention scores while others maintains, and the attention score between X3 and X4 grows larger enough than the significance threshold p. We call this attention adjustment successful "jump", and it could happen for the individual key.
We assume that there exists "miracle head", whose self-attention feature map contains both dot- product similarity and high-order statistics. Instead of modifying the linear transformation Wo direct "correct" the self-attention feature map by running thorough all triplets
and W K. and adjusting the scores by specific keys. However, this direct operation will introduce an extra computation of attention combination ((L)) w.r.t the input length L. And it is hard to acquire back- prokppropagation friendly implementation.
To formulate the modification of feature map, we relax the objective of finding the miracle heads to enhance the weak "jump" connection in canonical self-attention computation. Under the row view, build graph G (V. E) from the self-attention scores, where the queries compose the node set V with the row vector
as node features and each edge is defined by a adjacency matrix A. Then, introduce A 's computation. Subscribing the matrix with i7 returns the i-th column We select the j-th key as the current column, then the sub-adjacency matrix under the K, is defined:
vector.
A K U, - diag( U)ppp where U, S,T 2 S,S
(2)
d
The operation @ denotes the tensor product and we use the similar d-dimension normalization in the canonical self-attention [88. U is symmetrie matrix containing input pairs with (6, p, W Wx> similarity, like X3 and X4 in the previous example. [-]p is an indicator function for each matrix item that returns the item is greater than the p. otherwise return Here, we break the "both" requirements in Definition 3.1 Thus, the threshold p becomes hyperparameter reflecting the magnitude and p. We go through keys, the j-th aub-adjacency matrix reflects the similarity between different queries w.r.t K,.
[0] the summation of weighted adjacency matrix yield larger gap between isolated values and bulk eigenvalues, which has been widely used in helping the community detection task [14] 4 The sub-adjacency matrices describing the connections between all the queries, i.e, different layers in multiplex network, we can also leverage the same operation for a better edge probability recovery similarity detection. Then, we can add up all the sub-adjacency matrix into the weighted one:
Inspired by the layer aggregation
Σ (
Our next question enhance the "jump" connection for individual inputs. The Graph Convolutional Neural network (GCN) could perform local aggregate operators the graph adjacency matrix,
5

---

which is proposed by Scarselli et al. [S and is further simplified by Duvenaud et al. G and Sukhbaatar et al. [T6]. We employ it as a specific architecture for encouraging the attended items to become similar in both queries and keys. Then we have:
Q AQWO, K = AKWW
(4)
where Á = -- = + and D diagonal matrix from the column-sum of A, where every node can only attend its nearest neighborhood with the learnable graph filter W. The GCN will make the selected queries/keys propagate its features, which causes a higher score when encountering each other in self-attention computation. In other words, GCN encourages high-order self-attention's 's propagation through individual keys' similarity measurement. Thanks to this property, the self-attention score will be updated as:
(5) If we aggregate the graph filter W into the random projection W new projection W the score S' becomes A (XX))) ( k) J This could be consider as way modify the original learnable matrix We into the new projection W Note that the proposed method maintains the same parameter scale. Immediately, we could derive the self-attention score after applying the GCN operator on both queries and keys, namely the "jump" operation:
s' Q'K'T=
AX( X
(S)) = ÄSAT
(6)
which forms an approximated graph operator repeating propagation over the graph to enforce the desired triadic closure, like SGC (22]. The original GCN activation operation is omitted to avoid interfering with later self-attention calculations, which is merged into the following Feed-Forward Net. In this way, are able propose the Jump Self-attention:
($(QKT) Vd
A,Q.K. V) softmax
V
To better take advantage of the pre-trained model, we suggest to split H heads in canonical self- attention into two groups He heads and Hj heads. The He heads perform the canonical self- attention operation and the Hj heads perform jump self-attention, then concatenate their results as the complete layer output. The use of high-order self-attention enables acquiring high-order dependencies over the superficial ones. Overview of JAT's whole process is given in Appendix. The IAT and canonical self-attention can be used interchangeably in Transformer models.
3.2.1 Masking tricks
The mask of self-attention 18 plays an important role in avoiding the auto-regressive during the inference phase. The GCN operations may cause leftward information to leak between queries. If JAT applied in the decoder, we have to prevent this in building the adjacency matrix, where we reduce the edges from afterward queries previous queries. We implement this inside of JAT by forcing the upper triangular of adjacency matrix to in Eq.(3 The canonical self-attention's 's masking operation on the feature map is preserved after applying GCN in Eq. such that we can eschew the information leakage in softmax(.) +) function.
3.2.2 Efficient variants
The computation complexity of JAT remains O(L3) w.r.t the input length L, because we have to traverse all keys in the adjacency matrix aggregation of E3. We previously designed a sparsity measurement (25) to find the most significant queries in self-attention, which motivates us to employ it in selecting the primary keys:
M (K)) (()) (a)=mx(SS - mean(S,T)
(8)
We follow the sampling strategy and choose Top-u Keys. Thus, the original self-attention S reduces to S E IR f.x and we acquire the efficient variant JAT in the O(L)). Recalling that the JAT shares the same parameter scale, and the JATT holds the same computation complexity when compared with the canonical self-attention.

---

Table 1: The performance comparison the GLUE benchmark (dev sets).
CoLAA 8.5k
MRPC
RTF
STS-B 5.7k
QNLI 108k
QQP 363k
SST-2
WNLI 0.64k
MNLI 392k
Average
Model
3.5k
2.5k
67k
ELMo
44.1
76.6
53.4
70.4
71.1
86.2
91.5
56.3
68.6
68.7
BERT
56.3
88.6
69.3
89.0
91.9
89.6
92.7
53.5
86.7
79.7
BERT-4
62.3
90.2
73,9
90.1
91.1
90.6
92.9
55.6
87.3
81.5
BERT-IAT (ours)
61.6
89.1
73.3
90.5
93.2
91.6
93.5
57.3
85.6
81.7
BERT-JAT' (ours)
61.7
89.6
72.8
90.4
92.9
81.6
RoBERTatpce:
63.6
90.2
78.7
91.2
92.8
91.9
94.8
87.6
86.4
RoBERTa-JAT (ours)
65.9
91.4
80.2
91.3
93.2
91.7
95,4
87.7
RoBERTa-JAT' (ours)
66.7
91.7
80.9
92.3
92.9
91.7
94.9
87.5
87.1
JAT' represents the efficient variant of JAT. And the '-' indicates ahandoned experiments.
3.2.3 Higher-order variants
Without loss of generality, we can extend the similar in Definition 3.1 to a higher order case, where the similarity depends on more than input. Instead of performing analogical derivations, can introduce the Mix-hop [] GCN to capture the multi-hop propagation on the graph of queries. If we refer the canonical self-attention [88 to the first order self-attention and our JAT the second order one, then we can define the j-th order self-attention as:
S AJ S(A-1)T
if -
g (S)
(9)
where Á represents the adjacency matrix Á multiplied by itself (j times. It's trivial to substitute the 4 (S) in Eq. ( for higher-order self-attention variants _2,'4' and etc. We can split the heads into more groups when concatenating multiple self-attentions with different orders.
Experiments
In this section, we empirically demonstrate JAT's effectiveness on the pre-trained BERT BERTbaee base model with 12 layers, and perform BERT-JAT fine-tuning and evaluation ten NLP tasks.
4.1 Setup
(a) GLUE, We conduct JAT's experiments of the language understanding and generalization capa- bilities on the General Language Understanding Evaluation (GLUE) benchmark [21), a collection of diverse natural language understanding tasks. We also perform additional experiments on Su- perGLUE benchmark [20]. Settings: We use a batch size of 32 and fine-tune for 5 epochs over the data for nine GLUE tasks. The threshold p is selected from (4.0, 4.5...., 10.0). The other settings follow the recommendation of the original paper. Since the proposed JAT can be used interchangeably with canonical self-atiention, we perform grid search on the layer replacement. There are two sets of layer deployment, where the first combination is chosen from (Layerı-4 Layers-s, Layerg-12] and the alternative (L 12)) Another important selection the multi-heads grouping, we employ the "side-by-side" strategy as replacing the heads (2, 4, 6, 8, 10} with JAT. And we do not use any ensembling strategy or multi-tasking scheme in this fine-tuning. The evaluation is performed on the Dev set. Metric: We use three different evaluation metrics on the 9 tasks. Matthews correlation coefficient (MCC
TPxTN FPx
is used
N v: - yn POTP-FP)TTPPNNTNNPPPPP
is used for STS-B, and accuracy is used for others. Platform: Intel Xcon 3.2GHz The Nvidia V100 GPU (32 GB) X 4. The code is available at
for CoLA, Pearson (PC'C VN of [ ): /N Σ o
0z)a
com/ httphacyy//aa/22222222222
(b) SQuAD. We also evaluate on the SQuAD v1.1/v2.0
12 13
The settings is in Appendix D.
4.2
Main results
The result of the GLUE experiment is summarized in Table We compare EL g the baseline of GLUE, BERT (5), and RoBERTa (9), the pre-trained model where we build JAT varints. Our model BERT/RoBERTa-JA outperform base models on eight tasks without introducing more
7

---

Table 2: The performince comparison on the SuperGi.UH benchmark (dev sets).
Table 3: The performance on the SQuAD.
Model
Acc/F1
CB
BoolQ Acc
COPA
Acc
MultiCC FI/EM
WiC
Acc
WSC
Ace
RTE Acc
Average
Model
JSQuAD v1.1
FM
F1
SQQAAD v2.0
EM
F1
CBOW
71.4/49.6
62.4
63.0
20.M0.3
55.3
61.5
54.2
55.4
BIDAF-ELMO
85.6
63.4
66.2
M.FC
50.0/2.22
62.2
55.0
89.9/0.8
50.0
63.5
52.7
56.2
XLNetaase
89.7
95.1
87.9
90.6
BERT
94,6/93.7
77.7
69.0
70.5/24.7
74,9
68.3
75.8
75.8
BERT
81.2
87.9
75.9
79.3
RoBERTabse
92.8/93.7
81.5
74.0
70.7/28.4
69.1
64.4
78.7
75.9
BERT-A"
81.8
89.3
75.9
79.3
RoBERTa-JAT
70.2
65.4
80.2
BERT-JAT
82.1
89.3
76.4
82.0
RoBERTa-JAT
98.2/98.5
82.3
79.0
71.6/29.5
70.5
67.3 80.9
78.5
RoBERTa
88.9
94.6
86.5
89.4
JAT' represents the efficient variant of JAT.
RoBERTa- A
94.8
86.6
89.7
2 The indicates abandoned experimenss, and happeas when reaching out-of-memory.
RoBERTa-JAT
90.1
95.2
87.0
90.2
parameters, which shows that JAT can capture higher-order dependencies than canonical self-attention. More specifically, BERT-JAT achieve 9.4% score rising the CoLA dataset and 7.1 % on the WNLI dataset as a remarkable improvement, which shows that JAT shows the better alignment ability on smaller dataset. When it applies to RoBERTa, the average gain is larger and the most significant rising 4.9% happens on the CoLA dataset, which demonstrates a stable performance gain on SOTA models.
We have also show that JAT beats the competitor A 24 generally. The A introduces the triplet attention measuring dissimilarity to diversify the self-attention, while our proposed JAT tried to enhance similarity by building connections between unrelated pairs from different perspective. These two works are orthogonal and can be performed together to improve self-attention mechanism. Meanwhile, the efficient variant JAT also shows competitive performance than competitor A².
Results of evaluating JAT on the more challenging SuperGLUE benchmark are presented in Table 2 we have selected RoBERTa as a more efficient basic model. With more extended tokens on com- plicated tasks, the JAT fails on three tasks for OOM, but it still acquires better results than vanilla RoBERTa on others. The efficient variant RoBERTa-JAT" achieves the best average scores.
The result of the SQuAD experiment is summarized in Table Our model BERT-JAT achieves better performance than all baselines in both EM and F1 scores, which is aligned with the above results.
4.3
Ablation Study
4.3.1 Effect of JAT's Layer Deployment
In this experiment, we explore the JAT layer's s effect on acquiring the high-order connections. We set the layers at a different position in RoBERTabaee model to use JAT heads, where the RoBERTa- JAT represents that the layers from i to j in RoBERTaNese model have four JAT heads. We report the scores on the selected GLUE tasks, including CoLA, MRPC, RTE, and STS-B in Table 4 We can conclude that using JAT heads in lower or middle layers leads to better scores. This is consistent with the findings in Fig. 2). middle layers possess more weak high-order connections to be easily enhanced. Applying JAT at lower layers can effectively enhance the diversity of lower layers and improve the high-order connections force the desired triadic closure.
4.3.2 Effect of JAT's Head Grouping
To evaluate the effect of JAT's head grouping (different splits) on general language understanding task accuracy, we trained many RoBERTa-JAT models with different JAT head numbers in layer 1-6. Results on the selected GLUE tasks are shown T With the number of JAT head increasing, we can find that almost all the tasks' scores first increase then decrease. The scores increase because adding JAT heads enhance the high-order global dependencies in the vanilla RoBERTa model. They decrease when the number of heads is greater than 6, which probably is caused by too strong high- order inductive bias suppressing the canonical self-attention. But the CoLA dataset seems fitting the JAT assumption very well, which may explain why it reaches the highest performance gain in Table
4.3.3 Effect of Higher Order in JAT
We perform this experiment to evaluate the effects of higher-order variants. Following the similar settings as the previous ablation study, where the RoBERTa-JATT stands for the i-th order self- attention defined Eq.(9). (9. We show the selected GLUB results in Table And it reveals that for most natural language understanding tasks, the second-order self-attention is sufficient to discover
8

---

Table 4: The scores of different JAT's layer deployment.
Table 6: The scores of JAT's increusing order.
Model
CoLA
MRPC
RTE
STS-B
Model
CoLA
MRPC
RTE
STS-B BB
RoBERTa
63.6
90.2
78.7
91.2
RoBERTance
63.6
90.2
78.7
91.2
RoBERTa-JATmLA
63.9
90.9
78.8
91.2
RoBERTa-JAT
65,4
91.4
79.9
91.3
RoBERTa-JATggs [6.88
63.6
90.4
78.7
90.9
RoBERTa-JAT"
64.4
90.7
80.2
90.8
RoBERTaJJAT yp,
63.6
90.2
76.
90.9
RoBERTa-JAT
64.0
91.2
78.7
90.9
RoBERTa-JAT (1.0)
64.6
90.4
79.4
91.1
RoBERTa-JAT 63.8
89.7
78.4
91.0
CoLA Marthewss Correlation
0.725
Table 5: The scores of different JAT's heads grouping.
Model
CoLA
MRPC
RTE
STS-B
RoBERTabue
63.6
90.2
78.7
91.2
022
RoBERTa-IAT
63.7
91.4
80.2
91.1
12
0575
RoBERTa-JAT
63.9
91.2
80.2
90.6
RoBERTa-JAT
63.6
90.4
79.7
90.8
20
BERT-JAT
0250
BERTJATT
RoBERTa-JATas
63.8
90.2
78.9
90.7
1500031310 3364433
15:403:331:25 7
RoBERTa-JATan
64.1
90.2
77.2
90.5
Remaining Layers
Remaining Layers
RoBERTa-JATaa2
64.7
90.0
77.8
90.5
Figure 4: The performance decreases when the overall layers degrade from 12
more higher-order connections and improve the expression ability of the model, which leads to better results. For example, using the second order JAT on RTE dataset significantly improves the scores.
4.3.4 Effect of Layer Stacking Degradation in JAT
Similar to the evaluation in A 24 we conduct layer stacking degradation experiments on both CoLAA and RTE downstream tasks. Firstly, we reduce the simpler BERTbase model by one layer at time, which makes the overall layers degrade from 12 to 2. Fig. 4 shows that the BERT-JAT consistently outperforms BERT base- In the RTE task, the BERT-JAT even achieves the highest accuracy with eight layers. Secondly, we have extended the BERT base model from 12 layers to 15 layers, which matches the comparable extra computation brought by the JAT mechanism (refer to 4.6. It reveals that the deeper baseline lags behind the JAT-enhanced ones.
4.4 Visualization
We conduct visualization in Fig. 5 to show the self-attention feature map changing in JAT. To avoid repetition, we construct new sentence. The line between words represents their attention scores: the darker, the larger. The Fig. (5)) shows heads of layer in BERT and it indicates that the "Kate" has stronger connections to the words the second sentence. The Fig. (5b) shows heads of the same layer in BERT-JAT, the "Kate" attends to the first sentence's 's "hit" through "jump" connection with the blue head. This example shows that 'Kate' never gets a strong connection with the action 'hit' in vanilla self-attention, but JAT provides this conditional connection (may through "bite"), which indicates the two actions may be sequentially related. This JAT-inspired attention change also applies to the 'SEP' token, which implies that 'Kate' are the last few words. Although this result differs from the previous examples on the noun exchanging, it reveals that the Jump Self-attention can enhance connections with various lexical patterns.
4.5
Case Study
We conduct an experiment on Named-Fntity Recognition (NER) task using the standard CoNI.L-2003 dataset, which concentrates on four types of named entities: persons, locations, organizations and names of miscellancous entities that do not belong to the previous three groups. The settings are included in Appendix E. We select the major connections in the attention feature map where their scores are greater than (mean + std). We use 'CROSS' to represent connections of tokens with different -entedeettites. The visualization is showed Fig. F From Fig.(7b) we find there are not many "CROSS" connections in the attention of BERT base, and most of the connections are between tokens belong to person entity. From Fig. we observe JAT significantly increases the number of connections between tokens with different named-entities at the layers (1,2,3,8,9, having JAT heads, which enhances the model's ability to discover high-order information and makes the model's expression more diverse.

---

13:40
600
0
Training Time
Training Time
400
200
COLA RIE WMPC 515-8 The GLUE dataset
ONU
AKD C0
The GLUE dataset
(a) The faster group.
(b) The slower group.
(502)
(967)
(a) BERT base-
(b) BERT-JAT.
Figure 6: The run-time of the proposed Jump Self- attention (JAT, JATI) on the RoBERTa model. We divided eight tasks into faster (< 100 min) and slower group for clarity. Each bar stands for the training process on individual downstream tasks (running 10 epochs).
Figure 5: The sentence is "A friend of John hit the dog, then the dog bite Kate.", our BERT-JAT finds the "jump" connection between word "hit" and word "Kate" in 6-th Layer.
MISC
1.5
MBSC
14-
10C3
LOCK
CRO00
1.25
C1055
9.35
0.5
The Layar SelAAzorian
The Laye SelAlleiin
(a) BERT baxe-
(b) BERT-IAT
Figure 7: The pair-wise attention connections BERT base and BERT-JAT on dataset CoNLL2003, model BERT-JAT improve the connection between tokens with different named-entities (the 'CROSS' connections in orange).
Run-time Comparison
With the same setting, we evaluate the iterative efficiency of JAT on eight downstream task, which has no learnable parameters. The Fig. (6) shows that JAT mostly brings about 30% extra computation than the vanilla Transformer, where RTE achieves 75% at most. This variance is caused by the different input lengths when calculating the adjacency matrix in 33. It could be further improved by utilizing the Keys' sparsity (25) then JAT achieves no more than 15% extra computation time.
5 Conclusions
In this paper, discovered the existence of Jump Self-attention (JAT) in the Transformers' dot- product computation. Drawing inspirations from the pieces moving of English Draughts, we define JAT from the canonical self-attention and propose tangible framework to formalize JAT into the graph learning problem and leverage it to encourage JAT's propagation in attention feature map. We further develop the higher-order variants under multi-hop assumption to increase the generality. The proposed architecture is compatible with the pre-trained models. Experiments on two tasks demonstrate JAT's superior performance on capturing high-order dependencies. We also want to use JAT on MindSpore (https: /ww.. mi insporre... cn/) which is new deep learning computing framework.
Acknowledgments
This work was supported by grants from the Natural Science Foundation of China (62202029 and U20B2053). Thanks for the computing infrastructure provided by Beijing Advanced Innovation Cen- ter for Big Data and Brain Computing. This work was also sponsored by CAAl-Huawei MindSpore Open Fund. We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
10

---

References
[1] Sami Abu-El-Haija Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard. Kristina Lerman, Hrayr Harutyun- yan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In ICML, volume 97, pages 21-29, 2019.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, Language models are fow-shot learners. In NIPS, 2020.
[3] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's 's attention. pages 276-286, 2019.
[4] Aurelien Decelle, Florent Krzakala, Cristopher Moore, and Lenka Zdeborová. Inference and phase transitions in the detection of modules in sparse networks. Physical Review Letters, 107(6)006570,, 2011.
[5] Jacob Devlin, Ming-We Miig--Wei&ee Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. ACL, pages 4171-4186, 2019.
[6] David Duvenaud, Dougal Maclaurin. Jorge Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In NIPS, pages 2224-2232, 2015.
Rafael Gómez-Bombare BBombarell, Timothy
[7] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv:2101.0396,, 2021.
[8] Vidur Joshi, Matthew F. Peters, and Mark Hopkins. Extending a parser distant domains using few dozen partially annotated examples. In ACI. pages 1190-1 1999, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. 1rXiv:1907.11666 2019.
[10] Raj Rao Nadakuditi and M. E. Newman. Graph spectra and the detectability of community structure in networks. arXiv:1205.1811 1813, 2012.
[11] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In /CML, volume 80, pages 4052-4061, 2018
[12] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, pages 2383-2392, 2016.
[13] Pranav Rajpurkar, Robin Jia, and Perey Liang. Know what you don't 't know: Unanswerable questions for squad. In ACL, pages 784-789, 2018
[14] Joerg Reichardt and Michele Leone. (un) detectable cluster structure in sparse networks. Physical review letters, 101(7)008800,, 2008
[15] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Trans. Neural Networks, 20(1):61-80, 2009.
[16] Sainbayar Sukhhaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backprop- agation. In NIPS, pages 2244 2252, 2016.
[17] Dane Taylor, Saray Shai, Natalie Stanley, and Peter J. Mucha. Enhanced detectability of community structure in multilayer networks through layer aggregation. arXiv:1511.05271. 2015.
[18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008 2017.
[19] Jesse Vig. A multiscale visualization of attention in the transformer model. In ACL, pages 37-42, 2019.
[20] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general- language understanding systems. In NIPS, pages 3261-3275, 2019.

---

[21] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[22] Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In ICML, volume 97, pages 6861-6871. PMLR. 2019.
[23] Yang Yang, Jia-Qi Yang. Ran Bao, De-Chuan Zhan, Hengshu Zhu, Xiao-Ru Gao, Hui Xiong. and Jian Yang. Corporate relative valuation using heterogeneous multi-modal graph neural network. IEEE TKDE, 2021.
[24] Haoyi Zhou, Jianxin Li. Jieqi Peng, Shuai Zhang, and Shanghang Zhang. Triplet attention: Rethinking the similarity in transformers. In KDD, pages 2378-2388, 2021.
[25] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer long sequence time-series forecasting, In AAAL, pages 11106 11115, 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's con- tributions and scope? [Yes] We have include a complete Introduction section. Please refer to
Sec [
(b) Did you describe the limitations of your work? [Yes] We include that in the Limitations Broad Impact Section. Please refer to Sec.Appendix G.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We include that in the Limitations Broad Impact Section. Please refer to Sec.Appendix G.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theorctical results...
(a) Did you state the full set of assumptions all thcoretical results? [Yes] We have clarified in the Sec
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We provide all of them in the supplemental material and URL.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please refer See 4.1
(c) Did you report error hars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs. internal cluster, cloud provider)? [Yes] Please refer Sec
4. If you using existing assets (e.g., code, data, models) curating/relcasing new assets...
(a) your work uses cxisting assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? Ycs]]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're us-
ing/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A] All the dataset are public.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent
participant compensation? [N/A]
12


Reviews:
[{"review_id": "dcfMmGdC1rv", "reviewer": "Reviewer_MTPd", "summary": "The paper proposes a new self-attention mechanism that generalizes attention to a graph and takes into account connections with an intermediate node. The authors develop the intuition and frame their work in the context of prior work. They then perform experiments that show that their attention mechanism does indeed help the model perform better on more challenging linguistic tasks.", "questions": "They mention that their method increases computation by 30% or 15% depending on the configuration. That seems equivalent to adding an additional 1-3 layers to BERT. Does their method still perform well against a bigger BERT?\n\nSince their attention mechanism captures higher-order relationships, does it do well with on question-answering tasks that require multi-hop reasoning?\n\nWhat is the relative RAM requirements of their mechanism? How does it scale with L? The authors only address computation.", "limitations": "Possibly, their method becomes harder to apply on longer sequences.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The idea has roots in triplet attention and Graph Convolutional Networks which the authors mention. It does appear to be an original application to my knowledge.\n\nClarity: I like how the authors develop the intuition with the computer-pencil example. I was a little confused for some time because I expected to see a direct edge, but I think the authors meant they are connected only through the intermediate book token? The formal definitions and algorithms are clearly defined and motivated;\n\nQuality: Experiments are well done. I think there could be more tasks and performance benchmarks, though.\n\nSignificance: It's interesting to see that there does indeed appear to be better performance in the more challenging linguistic tasks, so I think the authors' intuitions seem to bear out and the method is efficient enough to be generally usable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "So8kgrIPMxE", "reviewer": "Reviewer_5zAj", "summary": "This paper proposes a method to model jump attention in sequence data (e.g., in the sentence “Jack is the father of John and John is the brother of Mary”, we have a jump connection between Jack and Mary using the stepping stone of John). Specifically, the authors add a one-layer GCN (without activation function) after a self-attention layer. The construction of the graph is based on the attention score (QK). In multi-head attention, parts of the heads are applied to the jump attention while the remaining preserve, making the network able to capture both first- and higher-order connection. To reduce parameters, the parameters of the GCN is merged into parameters of the self-attention layer. To reduce computation cost, the authors apply an existing sparsity measurement. The method is validated on GLUE and SQuAD comparing with baseline models and a recently proposed A3 method.\n\nThis is an interesting work with clear and convincing motivation, and it also reads smoothly. It is true that entities in texts may have relations without direct connection. The pivot entities (or say, stepping stones) become important under this case. Though GCN itself is not new, the definition of high-order connection (Definition 3.1 and Section 3.2.3) and the parameter-merging step in Eq (5) is novel. The authors find that the two parameter matrices of GCN and Attention can be merged (because they are both linear transformation).", "questions": "I have some questions for you. Correct me if I’m wrong:\n1.\tThe definition of the adjacency matrix A. From Eq (2), we can see if S_{ij} * S_{kj} is greater than the threshold \\rho, then i and k are considered as connected nodes. But this definition is a little bit different from Definition 3.1 saying both S_{ij} and S_{kj} need to be greater than \\rho. Can you explain more on this point?\n2.\tIn the combination of GCN and Self-Attention, the equations seem to omit the activation function of the GCN. Is it merged to the activation function of the later Feed-Forward Net? Or the ignoration makes no significant difference? It is better to elaborate the point clear in the method.\n3.\tWhat’s the effect of tuning the hyper-parameter \\rho?\n4.\tThe explanation of Fig. 4. I think it will be better if the definition of connection can be linked to some objectives or downstream tasks because it is hard to tell directly how “hit” is related to “Kate”. Also, in this example, it is unclear when we need jump connection or just first-order connection. For example in “The dog bite Kate”, is the “dog” directly related to “Kate” or they need a stepping stone “bite”?\n5.\tIn Line 272, “almost all the tasks’ scores first increase then decrease” may not hold since in Table 4, the main trend is just decreasing. I think it is better to add a RoBERTa model without JAT for comparison.\n6.\tHow slower it will be if not applying the sparsity measurement is shown in Fig. 5, but I wonder how much the performance will drop/increase.\n", "limitations": "Minor suggestions:\n1.\tThe writing can be further improved. There are typos in:\na)\tLine 121, “significant \\delta” should be “significance \\delta”.\nb)\tLine 162, “is a symmetric matrix contains input” should be “is a symmetric matrix containing input”.\nc)\tLine 176-177, “for encourage the attended items becomes” should be “for encouraging the attended items to become”.\n2.\tThe captions of Fig. 1 and Fig. 2 have large overlaps with your content. You can consider shrinking the captions to leave more space to your methods or related work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ Good readability. Same case study across the paper.\n+ Convincing motivation of jump connection.\n+ Novel method to combine GCN and Self-Attention.\n\nWeaknesses:\n\n- Lack information of comparison with related work.\n- Experiments on SQuAD have no results of the competitor A3.\n- Improvement is limited.\n\n\nUPDATE:\nThe authors' responses address my main concerns, which should be included in the revised version.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Rzzly-hxu2L", "reviewer": "Reviewer_mSLD", "summary": "In this work, the authors found that the high-order self-attention did exist in the dot-product computation but it was too weak to contribute much to the attention layer’s output. Take two sentences “Book costs more than pencil.” and “Book costs less than computer.” as an example. “Computer” never attends to “pencil” until they were both compared with “Book”, thus “Book” is the “stepping stones” in creating high-order attention between “computer” and “pencil”. In order to capture such high-order dependency in Transformer model, this paper proposed Jump Self-attention (JAT), and the main contribution can be summarized as:\n\n1. The authors proposed the JAT through the jump aggregation and high-order dependencies’ propagation without additional network parameters, and the efficient variant avoided the extra computation cost.\n2. The authors proposed the higher-order variants of JAT and the masking strategy to allow its interchangeable usage with the common self-attention on the pre-trained models.\n3. The experiments were conducted on ten different tasks to examine the effectiveness of the proposed methods. Results demonstrated the success of JAT in achieving state-of-the-art performance.", "questions": "The paper is clearly written. But, the game the authors mentioned was not popular.", "limitations": "The improvement is limited when compared with the related works.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The motivation of capturing high-order dependency was reasonable and very intuitive and the example they provided could help clarify concepts.\n2. This paper proposed the idea of capturing high-order dependency by measuring the similarity between different queries and keys, and then used the GCN to enhance this kind of connection. The idea is novel, and the result showed that the mothed could outperform canonical self-attention.\n3. This study designed several ablation studies to explan how their method worked. The run-time comparison was used to show the effect of efficient variants.\n\nCons:\n1. They said that they were inspired by the pieces moving of English Draughts, although they spent some space explaining the rules, However, this game is not so well-known, and may still cause some confusion.\n2. Although the efficient variants could indeed reduce the computation cost but still brought extra computation relative to the vanilla Transformer. And, the improvement from efficient variants to original self-attention was limited.\n3. This paper mentioned “miracle heads” several times, but the authors didn’t do comparison among these related methods. The authors may show the performance and computation cost of finding miracle heads.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "P6u3ZbXDdSJ", "reviewer": "Reviewer_YzKV", "summary": "This paper proposes Jump self-attention(JAT), which enhances first-order(i.e. dot-product of two hidden states) self-attention by running GCN\non its attention score map to capture high-order statistics. JAT can be interchangeably used with the canonical self-attention, and could be further generalized from second-order to higher orders. An efficient variant of JAT is proposed to avoid the extra computation cost. Experiments on GLUE and SQUAD show that JAT significantly increase the performance of BERT/RoBERTa-base with only 15% extra computation and no new parameters.", "questions": "## Notations\nDefinition 3.1 is confusing: are vector inputs x_i, x_j, x_k fixed vectors? If they are\nfixed, what does `Pr[\\dot]` mean? If they are random varaibles, what's the distributions?\nThe same question applies to weight matrix W_q and W_k.\n \nSection 3.2 is very hard to follow. For example, what is `adjacency matrix A` in #L159? \nIf it's the same matrix defined in Eq.(3), it's better to introduce that you're going to compute its components\nearlier instead of letting reader find it themselves after 10+ lines.\nI think a more detailed explanation of brute-force\nmethod in #L151~#L153 could help readers get better understanding on motivations\nand the derivation of matrix A in Eq.(3)\n\nThere are two `BERT-JAT(ours)` and two `RoBERTa-JAT(ours)` in Table1, what's the difference between them?\n\n## Experiments\nAn assumption of mine is that two layers of one-order attention \ncould simulate a single layer two-order attention. Therefore I'd like\nto see more experiments where authors use BERT/RoBERTa-large and \ntest whether the improvements of jump-attention still exist. \n", "limitations": " ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* This paper proposes novel and simple module to enhance performance of canonical self-attention with no new learnable parameters.\n* Experimental results show significant improvements over BERT/RoBERTa baselines, and authors provide ablation study on how to \ntune the hyperparams.\n\n\n### Weaknesses\n* Writing need improvement. The method part is quite hard to follow.\n* There is no experiments on BERT/RoBERTa-large models.\n\n\nI'd like to increase my score if the two weaknesses above are addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept