Global-aware Beam Search for Neural Abstractive Summarization
Authors: Ye Ma
Zixun Lan
Lu Zong
Kaizhu Huang
Venue: NeurIPS 2021 Poster (2021)
Keywords: beam search
global optimal
global attention distribution
summarization
sequence-to-sequence

Abstract: This study develops a calibrated beam-based algorithm with awareness of the global attention distribution for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search in a rigorous way. Specifically, a novel global protocol is proposed based on the attention distribution to stipulate how a global optimal hypothesis should attend to the source. A global scoring mechanism is then developed to regulate beam search to generate summaries in a near-global optimal fashion. This novel design enjoys a distinctive property, i.e., the global attention distribution could be predicted before inference, enabling step-wise improvements on the beam search through the global scoring mechanism. Extensive experiments on nine datasets show that the global (attention)-aware inference significantly improves state-of-the-art summarization models even using empirical hyper-parameters. The algorithm is also proven robust as it remains to generate meaningful texts with corrupted attention distributions. The codes and a comprehensive set of examples are available.

Global-aware Beam Search for Neural Abstractive Summarization
Ye Ma 14
Zixun L
Lu Zong 14-
Kaizhu Huang
Department of Financial and Actuarial Mathematics, School Science
Department of Applied Mathematics, School of Science 3 Department of Intelligent Science, School of A Advanced Technology Laboratory for Intelligent Computing and Financial Technology Xi' an Liverrool University, SIP, 215123 Suzhou, China
(yc.ma, lu.zong, kaizhu.huangl @ xjtlu.cdu.cn, 1 @tttudettt.iilluudduuuu.u
Abstract
This study develops a calibrated beam-based algorithm with awareness the global attention distribution for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search a rigorous way. Specifically, novel global protocol is proposed based on the attention distribution to stipulate how a global optimal hypothesis should attend to the source. A global scoring mechanism is then developed to regulate beam search to generate summaries in a near-global optimal fashion. This novel design enjoys a distinctive property, i.e., the global attention distribution could be predicted before inference, enabling step-wise improvements on the beam search through the global scoring mechanism. Extensive experiments on nine datasets show that the global (attention)-avy inference significantly improves state-of-the summarization models even using empirical param-parameters.. The algorithm is also proven robust remains generate meaningful texts with corrupted attention distributions. The codes and a comprehensive set of examples are available.
Introduction
As the barriers exist from the auto-regressive design of neural probabilistic text generators to pre- dicting the global optimum directly [30], the heuristic algorithm beam search that factorizes global optimization to multiple local optimizations, has been popularly used for text decoding [24]. In the original beam search setting, the global optimum among all possible sentences, and consists of words in vocabulary V. Given the global optimum step denoted as gst, the local optimums Ig refer K. candidate sequences with the highest probabilities at each step. While it is necessary to compromise on the beam size K. < V to ensure text quality [3, 25, 24] and search efficiency, beam search suffers from a major limitation due to its local property. Concretely, assuming that the global optimal hypothesis is within the K. local optimal hypotheses of the highest probabilities, i.e. p(gst) 2 p(lc.), for until the termination T, it operates solely with the local information available at each step. In practice, such assumption may however fail in the case that the probability of the global optimum at step T is less than those of the local optimums, i.c. p(gsr) < p(ler), ), but is adjusted to higher level in the later steps, p(g>7 (gsr) > ((Ir||l<>) and (p(sr (grr) > p(le, )p(s>//es This often leads beam search to get stuck in the local optimum from step onward in generating texts.
hypothesis g of the highest probability
corresponniing author wttp://cco/mmaammmmm/m/ /lcccbb aware
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Referance
Generated
Generated
Source
Source
Source
x1b
@lvdat Amendion Dommbbtton
Attertio
Atteniion Didiitutoon
Osimbttion
(8)
Alteston Drotttutton
Sourno trions
Bourco Kilena
Sounee tokeno
Deginning siage [local Grobel)
Tayy [LOOKE Geball
Terminatoo [LOCaI Giobel)
Figure 1: (a) Attention distribution is composed of the summation of cross attention the same- colored lines, distinguished from that different-colored lines which always equals 1 due to softmax. (b) Local attention gradually increases as the decoding proceeds. (c) Desired situation: growing local attention has been lower than global attention during decoding and exactly reaches at the end.
To cope with this limitation, this study proposes calibrated beam-based algorithm with global awareness at all searching steps. Generally, our novel algorithm is implemented in two phases. Before the beam search (Phase I), the global attention distribution predicted order be included as protocol to calibrate beam search at each step, encouraging the generated hypotheses to attend to the source in a more near-glow near-global optimal way. Specifically, the global attention distribution describes how all reference tokens should assign the attention to each source token (illustrated in Figure La), which could predicted from the source by training an irrention-predicttin model. The training fairly straightforward and resembles a sequence tagging task [12], except that the predicted attention distribution from the source a regression result. There are several advantages of using the attention distribution as the global protocol. 1) Attention distributions are sensitive to the decoder input, suggesting that any input to the decoder leads to a unique attention distribution with fixed model parameters; 2) attention distributions accessible for almost all text generation tasks thanks to the recent advances in attention models [27, 4, 31]; 3) relying on the source only, the global attention distribution can be predicted before beam search, thus offering a rigorous mechanism to calibrate global-award beam search.
During beam search (Phase II), we develop a novel global scoring mechanism composed of attention scores and length rewards guide beam search based on the predicted global attention distribution. As one main theoretical result, we show that the attention score can be considered as the probability that generated texts attend sources in line with the predicted global attention distribution. Specifically, the generated tokens in each step update the local attention distribution to source tokens dynamically, where the attention values grow monotonically as the generation proceeds (see Figure I.b). Since the desired situation is that the local distribution reaches exactly the global distribution at the terminal step, we regulate the inference by discouraging local attention from exceeding their corresponding predicted global attention at all steps.
With regards to the core target investigate the possible paradigm that improves beam search with global awareness during decoding, contributions of this study are summarized as follows:
We argue that the limitation of beam search roots from its defect in finding the global optimal hypothesis. We improve the algorithm by proposing global protocol to regulate beam search step-by-step. This paper is the first to predict and deploy the global attention distribution to calibrate the inference in a rigorous way, thus returning a hypothesis that attends to source tokens in more near-global optimal manner. In contrast, previous works [37, 7, 20, 29, 21] try to use attention distributions to improve beam search, but ignore that the global attention distribution is predictable.

---

A novel global scoring mechanism is designed to evaluate the generated sequences at each step based on the desired situation described in Figure I.c. As theoretically justified. its major component can be elegantly integrated into beam search in the form of a probability so that merely (IC) of the time complexity increased in each step (see Section $ 1.1 for more details).
The proposed algorithm with global awareness manifests a robust and plug-and-ple property in enhancing beam search for neural abstractive summarization. Without requiring any model or parameter modification, the global-aware inference shows excellent performance in generating meaningful texts, even if the attention distribution is cornupted not of its own. Further, it is identified that summaries generated by global-aware inference are both higher-quality and different from beam search hypotheses (see Global-aware in Table I). More interestingly, we find that the generation style of a dataset could be transferred by the designated global attention distribution. For instance, summaries of higher abstractness for CNN/DM could be generated by only replacing its global attention distribution with a highly abstractive distribution during inference. as presented in Global-awaret of Table 1.
On the empirical side, we show that the proposed global- aware inference can stably and significantly boost two state-of-th -f-the-rrt summarization models BART [17] and PEGASUS [39] to produce higher quality summaries on nine datasets, even if only the empirical hyper-parameten, are used.
Table 1: Use BART [17] fine-tuned in CNN/DM generate summaries. Global awbre the attention distribution learned from CNN/DM, while Global-avarei takes the attention distribution learned from XSUM.
Beam search
President Obama says climate change a public health issue that affects all of us. Obama: "No challenge poses more of public threat than climate change" Obama: "Millions people would lose their bealth insurance" if Affordable Care Act is not upheld. Obama: "I am not unticipating the Supreme Court would make such bad decision"
Global-aware President Ohama says climate change is public health issue that affects all of us He says the average American can their part to reduce their own carbon footprint. Obama did not appear particularly concemedd about Supreme Court challenge to the Affordable Care Act.
Global-awarei
President Barack Obama says climate change public health issue says the average American part to reduce their carhon footpeint.
2 Preliminary
The proposed decoding strategy is applied in BART [17] and PEGASUS [39] to perform sum- marization. BART is pre-trained seq-to-seq model whose structure essentially follows a vanilla Transformer encoder-decoder [31]. PEGASUS has a similar structure but pre-trained on a larger dataset differently. Notably, the fine-tuned parameters of both models are downloaded from Hugging- Face Models and are fixed in all subsequent operations, where "fine-tuned" means the pre-trained model has been fine-tuned specific dataset.
2.1 Beam Search
Since the decoder of the seq-to-seq model is auto-regressive model, the probability of target sequence y (Yo, ... yt.. ... MT) can be factorized to the probabilities conditional on the source (x1, Fit ... ,kn) and y<t = (yo.... "......V-- .Vt-1), i.e.
ply x) II (ollayye)) T21
When T 0, y (vo) and p(yolz) because yo is fixed start token. Beam search [8] is decoding strategy to predict a target sequence by maximizing this factorization. Given a vocabulary set V, at each inference step I, beam search selects a candidate beam set B = {b} 1 Jk= (where each
beam by: (b) ... by) is candidate sequence) from an all-possible beam set B, of size K. x [V] namely,
B. = {b; ou b E B,E,,, E VE}
(2)
hhaps:/huuginngtac..o/modss
3

---

attention B { b argtopk (logp(b, æ:), b, E B 1 121
(3)
where argtopk(-) outputs K. beams with the highest conditional probability, and o is the concatenation
operation. Besides, 3 = (bg k== where by is the start token. By Eq. 1, log p(bp x)) is an accumulated value. Its calculation can be simplified as:
log log((b;)z)) log p(b + p(ulz b²-1); t t22 2
(4)
pog) b)),
t = 1
where the value pogp((/111 (x) computed from the previous step. Therefore, at each step, we only need calculate the condition probability of each token in the vocabulary set.
beam is terminated after generates an ending token, and the beam set of terminated beams is defined as V. The final hypothesis y is chosen from y based on the beam probability normalized by of length [37]:
length where is a
log p(y az)
argmax y
(5)
(1
1)a
y/ y²
where y = (u6. ... w1)) 1 - 1 used since the start token is not considered calculating the length.
2.2
Attention Distribution
Attention distribution is a continuous vector whose element indicates the degree of attention paid to source token. The element is formed by the accumulation of cross attention, i.e., E, Oz,i, where oz refers to the cross attention that the t. token in the target sequence gives to the i.h source token." Specially, cross attention is a scaled dot-product [31] of hidden states of the source æ and the target sequence y. Notably, since Transdermerddecoder is an auto-regressive model, the assigned by 44 target token is actually calculated by V<r = (yo. -Mt-1)-
Global Attention Distribution. The global attention distribution g [91, - I.. 9n] R" is the attention distribution given by the reference, where global attention refers to the total attention that the reference attends to the ith source token, and is the source length.
Optimal Length. The summation of g, namely Ei= Is. is equal to Σ Σ -1=1 04,; T due to E Og,i = in softmax operation, where T is the reference length, or equivalently the optimal length Z.
Local Attention Distribution. The local attention distribution 1 1111,
L. ER is
the attention distribution of the kis generated sequence and updated each decoding step L. Thereinto, the local attention it, denotes the total attention paid to the ith source token by the kr beam sequence (b4, .......,f.. ... bb)
and is dependent on the sequence generated before f, i.e., bj (b) .)-))
3 Proposed Global-aware Inference
3.1 Global Scoring Mechanism
The global scoring mechanism consists of an attention scoring function and a length reward function. Given the global attention distribution g, the attention scoring function A(-) at the step depends on br_11
En
min(H,. 9i) 5t
A(b
t t2
(6)
7t
where ç: indicates the total attention that the generated sequence (bf, (b.) gives the source, and c = |bド_, / because the assignable attention for each generated token is 1. Notably, Eq. 6 attains the maximum score provided that each 4,, S gt. As mentioned in Section $ 1, the reason for
Mean pooling used for
and multi-heads

---

this design is that we desire the local attention iy, at the termination is exactly Ii, since the final hypothesis is expected to attend to source tokens the global-optimal manner. Meanwhile, we have lr, > 4, for 1. < T because 4,z monotonically increases on with a > 0. Therefore, at any step, the local attention 44, should not surpass 9t- Otherwise, the attention score will decline, and the penalty depends on the total amount by which these 4, exceed 9, (see Theorem 1). Further, the attention score could be considered as the proportion of correctly assigned attention to the total attention given by the generated sequence, where correct assignment indicates that all parts of l,, do not exceed g;. Also, it could be interpreted as the correct allocation probability of local attention against the global attention distribution (see Corollary 1.1). In this case, the total attention score be expressed as the same multiplicative form as Eq. to be elegantly integrated into beam search.
Theorem 1. Let M = {s ti.s > g-} and A = {6 Vs E M, ò = U1f 9o Given A = EsEr 8 where A 220 O, then A(b;11 decreases A increases.
Corollary 1.1. The bound of A(b)_1) is between 0 and
Proof. See App.A. A.
In addition to the constraint for h,i, still desire Ir, gi for each token. An ideal hypothesis should have two characteristics simultaneously. Namely, its attention score at the termination is the maximum and its length equals the optimal length. Therefore, we introduce a length reward function to cooperate with the attention score to penalize the situation (T,, # It. which will be discussed at the end of this subsection.
As mentioned before, the total attention score at the decoding step t is defined as:
A(b II
A(b
m==1
Thus, the joint scoring function J(-) is modified from Eq. 4: J(br, x)- log 1og (x)) + 9 log A(b + log p(v ar, b)
1-1
Σ
dogp(bmlas (mm)) 3 log A(b -(b=m=1)) 1)) + logp(u)aa b(-1) + log A(b;-1)
m=1
= (b 1: x) logp(u/ bt + 3 log A(b ,), t
and J(b) .)- = log p(u|2, b/)) + Bog A(b)), where 3 is hyper- hyper-paramen parameter to trade-off between the probability and attention score. Similar to p(b, (x) in Eq. 4, J(b,, x) is also an accumulative score. Consequently, at each step t, we only need compute p(rlæ, xx b brn)) and A(b(_1). Compared with Eq. 4, the time complexity of each step is only increased by O(K) as there are I. attention scores. Replacing log p( 6, (x) in Eq. 3 by (b. 2). we can select the top beams of each decoding step according to not only the probability distribution conditional on local information b but also the score conditional on global information g.
Considering the length reward function, the final hypothesis is thus defined as:
J(y y
y" argmax yy
B 2)
where R(-) is the length reward function dependent on the optimal length Z and the candidate hypothesis length ct. Exactly, 2 is the total attention that a candidate hypothesis (vi,... .....-,,)) pays to the source and equals 1y² 1. Besides, the attention score and length reward are weighted by param--ar 7, and the role of is ensure that the two are at the same level relative the probability. We remove in Eq. 5 as it only adjusts the length preference without really controlling the length.
The design of R(-) could be straightforwaad one only need ensure that it increases as ç approaches Z, and reaches the maximum only at + = Z. In this paper, we design step-wise length reward function R(C Z) to better balance the relationship between the attention score and the length reward and make the whole searching process as succinct as beam search. We put the design details of the step-wise length reward in App. B, and we regard Eq. 9 as the general scoring formulation of global-awe inference.
5

---

3.2 Predict the Global Attention Distribution
Since the reference is unknown practically, the global attention distribution could only be predicted from the source. We construct an attention attention-prediction model to learn the relationship between the source tokens and the global attention distribution.
The input of the attention-pret prretiionnreeiicioonn model the fixed encoder output E E R of BART or PEGASUS plus learnable positional encodings P E IRadd where d is the dimension of hidden states. The imput fed a learnable Transformer-encoder to obtain E Rnxd that encoded with additional context information, followed by a linear transformation with an exponential function:
g = exp EWW, + b,))
(10)
where 9 E IR" refers to the prediction of g. W d and b, E IR are the learnable weights and biases. The exponential function is imposed to ensure 9 > 0. We choose the exponential function for this operation because is shown stable in the training and testing stage. Given the objective of minimizing the distance between and g. the loss is defined as their Euclidean distance:
L g g
(11)
The predicted optimal length Ž is the sum of elements in J. Note that the length reward function is not affected no matter whether Z is an integer or not.
4 Experiment
4.1 Setup
Datasets. We evaluate the performance on totally 9 summarization datasets, where 2 datasets (CNN/DM [10], XSUM [5]) with BART [17] and datasets (XSUM [5], BillSum [15], Multi-News [6], NewsRoom [9], WikiHow [16], Reddit TIFU [34), arXiv and PubMed [2]) with PEGASUS [39]. Thereinto, XSUM [5] is a highly abstractive dataset whose summaries are all expressed a short sentence.
Implementation Details. We adopt randomly initialized 2-layer transformer-encode in the attention-prediction model wherethe structure of each layer is the same as the BART-encoder layer. attention- prediction
The optimizer is the Adabelief-optimizer [41] with eps 1e 16, betas (0.9, 0.999). weight decay le - 4 and learning rate 2e 5. The attention- prediction model is trained on the training set for about 50, 000 steps, and checkpoints are saved per 10, 000 steps to select the best checkpoints on the development set. Since the attention prediction is slightly different from common sequence tagging tasks, we have summarized two notable points after several attempts - the dropout rate should be and small learning rate is preferred. All experiments are conducted on 3 RTX 6000. We include the global-aware inference the generation code of HuggingFace transformers [36]. At the time of evaluation, ROUGE-1, ROUGE-2 & ROUGE-L (R-1, R-2 & R-L) scores [22] are computed from the ROUGE code used by BART [17].
attention-preddicion
Hyper-parameter Selection. Although the global -wwre inference requires two new hyper- parameters 7 and 3, some original typer-parameters of beam search, namely length penalty, minimum and maximum length, are omitted. The searching scopes of 33 and y are in {2, 4, 6, 10, 12, 15, 18, 20) and (0, 0.5, 1, 1.5, 2). respectively. According to the numerical tests on the development set, we finally choose 3 = 12, 7 1.5 for CNN/DM and 8 4, y O for XSUM. As limited improvement could be observed from larger 7 s, we recommend 7 = for normal or longer targets. When testing the global-aware inference with PEGASUS [39], we directly use empirical prper-parameters for each dataset, namely = 4, 7 = for XSUM and 3 = 12, ? = for other datasets. The beam size K. follows the setups in BART [17] and PEGASUS [39].
4.2 Results
Comparison with Beam Search. Beam search is hard-to-beal baseline which has stood the test of time and proven its superiority in practice for long [24]. In Table 2, we compare our global-aware

---

Table 2: ROUGE Fl scores of summaries generated by global-aware, in comparison beam search with length regularizations. Notably, global-aware uses empirical
XSUM R-2
BillSum R-2
Multi-News R-2
WikiHoww R-2
R-1
R-L
R-1
R-L
R-1
R-L
R-1
R-L.
Beam search [39]
47.05
24.53
39.33
57.00
39.65
52.70
47.29
18.91
43.31
41.86
19.04
40,40
Global-aware
47.33
24.66
39.50
58.66
40.12
53,96
47.95
19.08
43.93
42.82
19.68
41.43
Reddit TIFU
NewsRoom
PubMed
arXiv
R-2
R-L
R-1
R-L
R-1
R-2
R-L
R-2
R-L
[39]
27.55
42.05
44.25
Globul-aware
28.31
9.13
23.30
44.68
31.71
41.28
45.78
20.16
42.62
44.92
17.41
40.31
inference to beam search with length regularizations (i.e., a in Eq. 5, accompanied with two hard constraints, namely minimum length and maximum length). We strictly follow the hyper-parameter setups of PEGASUS [39] in terms of beam search, while we only adopt empirical hyper-parameters for our method. Even so, significant improvements can be observed on all the data sets, especially when the summary is of normal or longer length.
Table 3: Comparison with other methods.
Table 4: ORACLE and ablation results.
CNN/DM R-2
CNN/DM R-2
K
R-1
R-L
R-L
Beam search [17]
44.12
21.21
40.89
ORACLE global-aware
51.85
28.13
48.68
Our coverage
44.74
21.69
41.48
wWo length reward
50.46
27.53
47.43
Repetition penalty [13]
44.11
21.14
40.87
Attention masking
45.54
22.24
42.44
Global-aware
-wfo length reward
44.39
21.58
41.41
Global-aware
45.13
21.77
42.04
-wfo attention score
21.29
40.91
XSUM
XSUM
K.
R-1
R-2
R-I.
R-1
R-2
R-I.
search [17]
45.38
22.32
ORACLE global-aware
26.
Our coverage
44.54
21.82
36.97
-w/o lengch reward
48.92
26.48
41.45
Repelition penalty [13]
45.40
22.31
37.13
Attention masking
45.35
22.31
37.15
Global-aware
45.44
22.15
37.11
-wlo length reward
45.57
22.60
37.61
22.60
37.61
Table 5: Improvements of attention head masking and global-aware on beam search [39] in terms of ROUGE-L F1 score. Both use empirical setups.
XSUM
BillSum
Multi-News
WikiHow
Reddit
NewsRoom
PubMed
arXiv
Attention masking Global-aware
-0.31
0.23
0.35
0.17
1.26
0.62
1.03
1.18
2.58
1,51
1.03
Comparison with Other Attention Approaches. In this part, we focus on comparing other ap- proaches which also exploit certain attention distributions to improve beam search. The first is the coverage penalty [37, 21]. To enhance its performance in summarization, we replace its pre- set attention distribution with our predicted global attention distribution. Note that the coverage function can only evaluate the generated sentences at terminal step. Instead of comparing the global-aware inference to the methods [29, 20, 7] that aim to reduce repetition using the dynamic attention distribution, we compare our algorithm with the CTRL repetition penalty [13] which has similar motivation but is more systematical and independent of training. Table lists the comparison results against different algorithms. It can be observed that our global-aware approach can improve the performance of beam search stably and fundamentally. We also observe that the attention head masking [1] appears to outperform the global-aware approach on CNN/DM, but it fails to gain any improvement XSUM. To further show the advantage of the proposed approach, closer examination on the attention head masking [1] and our proposed approach the next part.
will take
Further Comparison with Attention Head Masking, First, one should bear in mind that the attention head masking [1) acts on the model instead of beam search, which is opposite to us. Specifically, it selects contents during decoding by disabling partial attention heads for unimportant

---

Table 6: Generate CNN/DM summaries with XSUM's style.
Figure 2: Sensitive analysis in the test set.
CANDM
CNNDM,.Hfg
R1/R2/RI
Shorter beam search
Global-awarei
RLLI 2000es
-1.5
F1 score
43. 43.6/20.940.44
43.6/20 4/40.6
Recall
48.9/23.4445.2 9//3.1
00.4//8.8/377
Precision
50.5/23.8/47.00
Table 7: Generate summaries with corrupted attention distributions.
42.5
XSIM
XCL.M, B-A
37.6
RL soCitC
R-1
R-2
R-L
07.4
37.4
Beam search
27.00
12:21
23.88
13:22
Globul-aware (10K)
28.78
12.82
25.51
Global-aware (100K)
29.59
13.72
26.30
37
37
tokens to decrease the attention these tokens. According to the reported results presented in Table 3, we can see that although attention masking achieves amazing results on CNN/DM, it does fail completely on XSUM. Since hiding unimportant tokens from some heads results in the loss of context information of salient tokens, this would lead to its instability. Thus, it could be ineffective for tasks that require contextual information of the whole source such as XSUM. Taking further comparison, we deploy the attention head combinations selected for CNN/DM and XSUM to examine its effect on PEGASUS [39]. These comparison results are shown in Table 5. Evidently, our method enjoys robust feature that is able to boost summary inference on various datasets and models even with the same set of apper-parameters In contrast, attention masking [1] behaves much sensitive to the changes of models and datasets. Besides, attention masking has to construct its training saliency labels based on the longest common subsequences between a reference summary and the source. This may be hardly achieved in some text generation tasks (e.g. translation) where no common subsequence exists at all. Such drawback presents one main limitation for attention masking.
ORACLE Results and Alltion Study. ORACLE refers to the global-aware inference combined with (true) global attention distribution instead of predicted global attention distribution. The related results have been presented in Table 4, and the significant boosting certifies that the proposed method could improve beam search with the global attention distribution. On the other hand, we conduct ablation experiments on ORACLE and global-aware. Both results indicate that length reward plays an important role in generating normal-length text but causes adverse effect on generating text of very short length. Besides, the performance declines significantly when only length reward is applied, due to the fact that sole length reward cannot calibrate beam search step-wise.
Robustness. We intend to examine the robustness of our proposed global-aware algorithm in this and next part. To do so, we substitute the parameters in CNN/DM's attention prediction model to XSUM's to create more abstractive summaries for CNN/DM. For comparison, we set the minimum length of beam search as to allow it to generate shorter summarics. Table 6 shows the Fi score, recall and precision of the shorter beam search and global-awaref. It is surprising that even by using the attention distribution from different dataset with distinct properties, the proposed global-awan -ware mechanism still manages to generate meaningful summaries with competitive F1 scores, proving the robustness of this algorithm. Moreover, the higher Precision and lower Recall of the global-awar -aware suggest that although information is partially lost, the algorithm still summarizes core information in concise format, compared to the standard beam search. On the other hand, we exploit a BART model fine-tuned on CNN/DM to generate summaries of NewsRoom directly, and the ROUGE scores of beam search are shown in Table 7. Next, we randomly select 10K and 100K samples from the training set and use them to fine-tune the attention-prediction model, where the global-awar improves beam search substantially. The experiment once again validates the robustness of the proposed inference algorithm, as it maintains a reasonably good performance even from learning corrupted attention distribution from a BART without muningn
Sensitive Analysis. We further examine the performance robustness with respect to the hyper- parameters. From Figure 2, we could see the global-aware inference is always better than beam search in CNN/DM, no matter how its hyper-parameters change. Besides, the performance is less sensitive to the hyper-parameters when 3 2 10 or 7 2 1. While XSUM, the global-aware could improve beam search stably with 7 = O, but there is significant decline when applying length
8

---

(e)
(B)
(0)
Global Altention
predicted pradle
Global Altention
Global Altention
0.6
0.4
100
200
300
400
1.00
200
500
400
100
200
300
400
Source
Source
Source
Figure 3: Predicted and ORACLE global attention BART. There are attention distributions of (a) the whole source, (b) the source without the start & end tokens, (c) the source without the start & end tokens and full stops.
(e)
(b)
(c)
Attention Charges
Attention Changes
Atlention Changes
0.4
0.04
0.1
0.2
0.02
-0.02
200
400
600
200
400
600
0
200
400
600
Source
Source
Source tokans
Figure 4: Changes of the attention distribution when (a) one word in the reference is replaced by a similar word (sl) and a random word (s2), (b) the sentence order of the reference shuffled, (c) factual knowledge in the reference is distorted.
reward. fact, the attention score favors shorter hypotheses, and the length reward could alleviate the bias. However, if the references of dataset are already very short such as XSUM, the length reward may lead to a counterproductive effect. Since the setup of CNN/DM is applicable to most datasets, we argue that the global-aware inference robust both hyper-parameters in most cases.
Analysis on Global Attention Distribution
5.1 Distribution Law
The distribution law of global attention in BART [17] is shown in Figure 3. It is observed that most attention is assigned to the start token and full stops which are not semantically salient, and most of the remaining attention is unevenly allocated to (semantically) overlapped tokens between the source and the reference (i.e., important words). It is worth mentioning that the importance here is longer a simple binary classification like in [7, 11, but a continuous numerical value decided by the model knowledge learned from data. In general, with the word importance, but should be clear that it essentially reflects knowledge learned by the attention model such the syntactic and semantic structure of sentences. Meanwhile, the distribution law indicates that attention distributions in pre-trained models may not be relevant with the uniform distribution all. That is say, it is not reasonable still use an uniform attention threshold (like the threshold preset in [37, 7J) to regulate the decoding of pre-trained models. Last but not least, our general motivation is alleviate locality bias by aligning the attention distribution of reference and hypothesis, which does not really care how the global attention is distributed only if it is predictable. However, the proposed penalty mechanism is indeed insensitive some distributions, and we will provide more thinking about this in App. G by applying the global-awar inference to translation.
should not simply equate the global attention
5.2
Why It can be Predicted from Source?
Since the ORACLE experiments indicated that the global attention is helpful for decoding, the concern remains if it is predictable using only source. In App. E.1 we will show the predictability empirically, while here we just provide an interesting explanation. In our opinion, the global attention distribution is interpretable representation of reference which not only has the characteristics of hidden representation also can be explained by the source tokens. First of all, given the source, the global attention is calculated by the input reference and trained neural network parameters;

---

this is similar to achieving hidden representation. Moreover, like hidden representation, the global attention distribution could also capture the semantic similarity, e-g.. replacing a reference word with a semantically similar word leads to slighter attention changes than that with random word (see Figure 4.a). Besides, it is observed from Figure 4.b and Figure 4.c that global attention able to represent the sentence order and factual knowledge. On the other hand, the global attention distribution enjoys a distinct characteristic that each of its features can be explained as the degree of attention paid to source token, which means changes of such representation are regular to some extent. For example, in Figure 4.c, we distort a numerical number in the reference to violate the fact stated in the source, and then find that the attention assigned to the actual number originally is most transferred to the special tokens and punctuation. Overall, similar sources should have similar global attention distributions, since similar sources often have similar references and global attention distribution is a representation of reference. Moreover, the global attention and source tokens are in an one-to-one correspondence. Thereby, we argue that it is convenient to use the source to predict the global attention distribution.
Related Work
In the field of text generation, efforts have been made to boost beam search with regards to the attention distribution. For instance, some studies engage the attention distribution to penalize the situation where sources fail to be fully covered in translation tasks [37, 21]. while others [7, 29, 20] incorporate dynamic attention distributions to evade tokens that have been highly regarded to reduce repetition. However, none of the aforementioned studies attempts apply the global attention distribution to acquire the knowledge that the level of attention should be allocated to each token from the reference. Further, the existing score functions used by those studies are rather different from the proposed global scoring mechanism. More details can be seen in App. D.
In addition to the attention distribution, other techniques are developed to improve beam search in terms of length bias [38], diversity-less [32], vapidity [11] and degradation [25, 3). These methods are not included for comparison because they are not suitable for summarization [38, 11], or do not aim to enhance beam search as the main purpose [32, 25, 3]. Besides, we argue that these patches to beam search are supposed to be hard for improving the performance stably and fundamentally (as shown in Table 3 given by our global-aware method) because they fail to specify what a final hypothesis should look like and are easy trap into local optimums.
7 Conclusion
Beam search tends to fall into local optimums due to the lack of global information during inference. To calibrate beam search on the premise of global awareness, this study proposes a predictable protocol to stipulate how global optimal hypothesis should attend to source tokens. By training a simple prediction model of the global attention distribution, a novel global scoring mechanism is then developed to regulate the inference on a step-wise basis. Our experiment shows that the proposed global-aware beam search generates higher-quality summaries, relative to beam search and other counterparts of the similar nature. Besides, the proposed algorithm is proven robust generating meaningful texts with corrupted attention distributions, implying its potential cooperate with user-defined global attention distributions. We plan to focus our future study on generalizing the global-awar inference to broader range of text generation tasks, including not only text2text but also image caption [33], multimodal summarization [19], graph2text [14] and data2text [26].
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: XITLU Key Programme Special Fund KSF-A-14 and KSF- P-02, National Natural Science Foundation of China under No. 61876155 and Jiangsu Science and Technology Programme under No. BE2020006-4.
References
[1] Shuyang Cao and Lu Wang. Attention head masking for inference time content selection in abstractive summarization. In Proceedings of the 2021 Conference of the North American
10

---

Chapter of the Association for Computational Linguistics; Human Language Technologies,
pages 5008-5016 2021.
[2] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim. Walter Chang, and Nazli Goharian. A discourse-awand attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages 615-621, 2018
[3] Eldan Cohen and Christopher Beck. Empirical analysis of beam search performance degradation in neural sequence models. In International Conference on Machine Learning, pages 1290-
1 1299, 2019.
[4] Jacob Devlin, Ming-War Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 1186, 2019.
[5] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language un- derstanding and generation. Advances in Neural Information Processing Systems, 32:13063- 13075, 2019.
[6] Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large- scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074- 1084, 2019.
[7] Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 919984109,, 2018.
[8] Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211 1.3711, 2012.
[9] Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pages 708-719, 2018.
[10] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28: 1891-691770, 2015.
[11] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. text degeneration. In International Conference Learning Representations, 2019.
The curious case of neural
[12] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-erf models for sequence tagging. arXiv preprint arXiv:1508.0199: 2015.
[13] Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. 2019.
[14] Rik Koncel-Kedziorski Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. Text generation from knowledge graphs with graph transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologics, Volume (Long and Short Papers), pages 2298-2293, 2019.
[15] Anastassia Kornilova and Vladimir Fidelman. BillSum: A corpus for automatic summarization of US legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 48 48-56, 2019.
11

---

[16] Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. arXiv preprint wrXiv:1810.09305 2018.
[17] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, 2020.
[18] Jiwei Li, Will Monroe, and Dan Jurafsky. Learning to decode for future success. arXiv preprint arXiv: 171.069999.6449 2017.
[19] Mingzhe Li, Xiuying Chen, Shen Gao, Zhangming Chan, Dongyan Zhao, and Rui Yan. Vmsmo: Learning to generate multimodal summary for video-based news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9360 00009936922200 2020.
[20] Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. Improving neural abstractive docu- ment summarization with structural regularization. Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4078-4087, 2018
[21] Yanyang Li, Tong Xiao, Yingiao Li, Qiang Wang, Changming Xu, and Jingbo Zhu. A simple and effective approach to coverage-aware neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
pages 292-297, 2018.
[22] Chin- Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, 2004.
[23] Yinhan Liu. Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.082. 100200,,,00........0..0. 2020.
[24] Clara Meister, Ryan Cotterell, and Tim Vieira. beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173-2185, 2020.
[25] Myle Ott, Michael Auli, David Grangier, and Mare" Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3956- 3965, 2018
[26) Ratish Puduppully, Li Dong, and Mirella Lapata. Data-to-text generation with content selection and planning. In Proceedings of the AAAI conference on artificial intelligence, pages 6908-
6915, 2019.
[27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
[28] Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv: 151 1.06732,672 2015.
[29] Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, 2017.
[30] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112, 2014.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6000 2017.
[32] Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In Thirty- Second AAAI Conference on Artificial Intelligence, 2018.
12

---

[33] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[34] Michael Völske, Martin Pouthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Red- dit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, 2017.
[35] Sam Wiseman and Alexander M. Rush. Sequence-to-sepee -o-suueecc learning beam-search opti- mization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2296-1306, 2016.
[36] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, 2020.
[37] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, et al. Google's 's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08 1444, 2016.
[38] Yilin Yang, Liang Huang, and Mingbo Ma. Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054-3059, 2018.
[39] Jingqing Zhang. Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR, 2020.
[40] Wen Zhang, Yang Feng, Fandong Meng. Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334 4343, 2019.
[41] Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in Neural Information Processing Systems, 33, 2020.
13


Reviews:
[{"review_id": "zNTf-MLwrlf", "reviewer": "Reviewer_QnKK", "summary": "This paper tackles the problem of suboptimality of beam search as it is not able to have a global perspective of the probability of the generated beams due to truncated beam size (less than vocabulary size) for space and time complexity purposes. This paper proposes a global-aware beam search which is calibrated using the global attention distribution to guide the local decisions at every time step during beam search. The global attention distribution prediction is formulated as a regression task, which is then fed into a global scoring function to guide the beam search-based generation. This approach solves the mismatch between the local optimality of beam search with the global optimal generation. This modified search strategy has been shown to boost the summarization of two SOTA models BART and PEGASUS.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points:\n1. The paper is well-written and clear.\n2. The problem is well-motivated and theoretically sound.\n3. The idea of using attention distribution to guide beam search in a principled manner is novel and interesting.\n4. The experimental result improvements are convincing across all 9 summarization datasets.\n5. Thorough analysis has been done to compare with beam search across 9 summarization datasets, in addition to comparison with other methods like coverage, repetition penalty, attention masking, along with ablation studies across various dimensions.\n6. The proposed algorithm has also been shown to be robust by using attention prediction from a different dataset.\n\n\n## Questions:\n1. It is not clear (or maybe there is some gap in my understanding), whether the predicted attention distribution still leads to the same theoretical result (Theorem 3.1 and Corollary 3.1.1)\n2. It will be interesting to see how this decoding strategy fares in other NLG text2text tasks like Paraphrasing, Neural Machine Translation, etc.\n3. How does the proposed strategy compare with other decoding strategies like Top-k Sampling[1], Nucleus Sampling[2]?\n4. For completeness, human evaluation with respect to correctness, adequacy, fluency, and coherence will be a helpful comparison.\n5. The related work can include more guided decoding strategies as summarized in the blog[3].\n\n## Originality: 7/10.\n\n## Quality: 8/10.\n\n## Clarity: 8/10.\n\n## Significance: 7/10.\n\nREFERENCES:\n[1] Hierarchical Neural Story Generation, Fan et al.\n[2] The Curious Case of Neural Text Degeneration, Holtzman et al.\n[3] Controllable Neural Text Generation: https://lilianweng.github.io/lil-log/2021/01/02/controllable-neural-text-generation.html\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "vEIYC4Aq_W", "reviewer": "Reviewer_ke1n", "summary": "This paper presents a method to improve beam search by learning and predicting the global attention. A global scoring function is developed to regulate beam search to generate summaries. In Phase I, the global attention distribution is predicted in order to be included as a protocol to calibrate beam search. In Phase II, a step-wise global scoring function is used to guide beam search based on the predicted global attention distribution. \nExperiments on 9 datasets show that the global-aware inference improves state-of-the-art summarization models. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The angle and direction is pretty interesting. While most of prior work on improving beam search focuses on repetition, similarity/dissimilarity, and length control, this paper presents a relatively new angle. The method proposed can implicitly improve the content planning procedure of text summarization and generation. \n\nThe performance improvement is significant. Comparing with other methods (Table 4), it is also competitive. \n\nQuestion:\nLine 96 - 98: “With both sources and  reference targets available in the training set, the attention given by reference tokens is computable, called global cross attention αg ”\t\t\t \t\t\t\t\n\t\t\t\n\t\t\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Presentation \nAlthough I believe the idea of this idea is actually pretty straightforward, the presentation and illustration of the idea are hard to follow. There are just so many notations, superscripts and subscripts, like alpha {l,k,t,i}, zeta, G, etc. I believe the presentation can be significantly improved. I would like to see a figure with a toy running example, which I think will walk through how the method works, during training and during inference. Something like source = A B C, decoding prefix = y0 y1, the local attention distribution is [0.1, 0.2, 0.7], the global attention is [....],try to minimize the distance of __ and __, during inference, step 0, step 1, etc. \nIf the authors want to convince more people to use this method, this step is necessary, from my point of view. \nThe naming and symbols could be simplified like “local cross attention as the dynamic local attention distribution”.\n\nThe method is a little hacky and complicated, and I would like to see more analysis about how attention changes, what’s the global and local attention. The evaluation mostly focuses on ROUGE, but I still don’t have much idea about what happened and what’s the change of the attention pattern or behavior. If this piece can be shown, I believe the paper will seem less hacky because you explain how it works. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "mol7hbKF5_v", "reviewer": "Reviewer_M9Ny", "summary": "This paper thinks that cross attention distribution in transformer’s encoder-decoder framework can help the performance of beam search. At the same time, this paper also proposes an algorithm to predict distribution to help beam search. This paper conduct experiments on 9 datasets to prove its effectiveness.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There is no doubt that the empirical results given in this paper are quite strong, but it is difficult for me to understand every detail of the paper and the motivation of the paper.\n\nIs there a possibility that the performance improvement is obtained by the increase of model parameters for attention prediction? Usually, better performance can be obtained by increasing the model parameters.\n\"\"\"\n197 Implementation Details. We adopt a randomly initialized 2-layer transformer-encoder in the\n198 attention-prediction model, and the structure of each layer is the same as the BART-encoder layer.\n\"\"\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see review", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "d8ZQzQd5B4E", "reviewer": "Reviewer_eHvy", "summary": "This work proposes to add a global attention feature component to the beam search scoring procedure for conditional generation, which typically just considers the fitness of successor tokens conditioned on the predicted history of tokens in the beam. This augmentation basically keeps track of cumulative local autoregressive attention on the source tokens during beam search and penalizes major deviation from the predicted desired \"global\" attention over the source tokens. Specifically, this approach proposes a finetuning step of neural language generators, which is responsible for predicting cumulative global attention on each token and the training is done via regression. During training, the \"reference\" global attention is computed by using the full reference in the decoder transformer instead of just the left prefix of the reference to compute attention.\n\nA relatively simple scheme is proposed to measure deviation of cumulative local attention from the global attention in a decomposable iterative manner. This scheme also involves a form of length reward that encourages the generation to match the length predicted by the global attention.\n\nExperiments are performed on the task fo summarization with pretrained summarization system on 9 datasets. The results show that the proposed beam search augmentation outperforms standard beam search in general. Most strikingly, the ablation experiment with the \"Oracle\" global attention (instead of prediction, if we had access to the actual global attention and hence the actual reference length) significantly outperforms beam search and the proposed approach demonstrating the potential of global attention.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n\n-- The proposed approach is well motivated, very simple to implement and novel to my knowledge.\n\n-- The results on abstractive summarization show that the proposed approach is effective. Specifically, the oracle experiments clearly show that global attention, although not used during training of the seq2seq models, is a useful signal to model.\n\nNegatives and Questions:\n\n-- The experiments are only on the task of summarization where the token overlap is huge. I would have liked to see the performance of this approach on other related tasks like machine translation.\n\n-- It took me several readings to figure out what exactly was trained/finetuned--and I am not completely sure even now. I believe that the seq2seq models were pretrained and used local attention as is standard during training. The architectures were finetuned to predict global attention. While it is clear that finetuned models were used for global attention prediction, were they also used for decoding or were the pretrained models used for performing the beam search.\n\n-- It is surprising that global attention would be useful given that it is not at all used during training of seq2seq models. Is it mainly useful because of token overlap between source and reference--i.e. while training global attention can use reference to tell the decoder which source tokens are important for the future steps? Again, this is a question that would be better answered by experiments on another task like MT.\n\n-- On the second point, while the proposed approach is compared to beam search, I think there are two beam search baselines to compare to. 1) Beam search with the pretrained BART/PEGASUS, 2) Beam search with finetuned BART/PEGASUS.\n\n-- Also, what happens if the seq2seq training includes global attention (or the full seq2seq training is finetuned with global attention)? This would mean that in addition to finetuning with the regression loss, you would perform finetuning with the actual seq2seq cross entropy loss with global attention.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I found the sparse discussion sufficient given the mostly algorithmic contribution of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "-kD6XuA5ys", "reviewer": "Reviewer_9iis", "summary": "This paper proposes a generalization of the attention coverage penalty in seq2seq models (Wu et al 2016), where instead of encouraging a uniform coverage of the source, this work first predicts the expected coverage of each source word. During beam search, beam hypotheses that violate the expected attention coverage constraints receive a penalty. Experiments on 9 summarization benchmarks show that the proposed coverage loss outperforms using existing attention coverage losses during beam search.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This work generalizes the attention penalty loss, where different variants are used in different applications (for example, translation often encourages uniform coverage of the source, whereas summarization often penalizes repeatedly attending to the same word to reduce repetitions). In theory, the expected attention coverage predictor can learn to vary its behavior according to the task, instead of relying on the expert design of the form of coverage loss.\n2. The approach proposed by this paper can be viewed as maintaining similar statistics between training and testing (similar to professor forcing), and this might inspire future research matching other statistics such as the distribution of hidden states.\n3. Experiments show improvements over existing coverage mechanisms.\n\nWeaknesses:\n1. In Table 4, the last row shows that Global Aware has R1, R2, RL of 45.57, 22.60, 37.61, but according to table 3, this is the performance of Global Aware without length reward, whereas the performance of Global Aware in table 3 is 45.44, 22.15, 37.11.\n2. While the proposed approach might alleviate the locality bias of beam search, it still suffers from a similar locality bias. The attention constraints (Eq 7) only become active when the accumulated attention exceeds its total budget, which I expect to happen rarely at the beginning of the search (when accumulated attentions are small) and more frequently towards the end of the search (when accumulated attentions are larger). Therefore, I wouldn't expect this \"global term\" to guide early decisions during search, and this is exactly where beam search has lots of uncertainty and global planning is needed the most. For this reason, I don't think the proposed approach (or existing attention coverage penalties) is truly \"global aware\" unless there's some estimate of future rewards (https://arxiv.org/pdf/1701.06549.pdf).\n3. In practice, the proposed approach introduces additional components: an expected total attention predictor, as well as additional hyperparameters. While using empirical hyperparameters for some settings results in good performances, I'm not totally convinced how robust the performance is with respect to hyperparameters, and how much tuning effort is needed. \n4. The comparison to the baseline coverage in table 4 does not appear fair. According to my understanding coverage doesn't include length penalty, and if we compare baselines in table 4 with Global Aware without length reward in table 3, coverage underperforms on XSUM but outperforms on CNN/DM, and the other baselines also appear to have similar performances to the proposed method.\n5. In Figure 2, the performance changes are not significant, and I think using even larger beam sizes would show a clearer trend.\n\nQuestions for authors:\n1. How well does your expected attention predictor predict attentions? Is it predictable just from the source? Wouldn't different summaries for the same document pick source words differently?\n2. In Eq 8 why not directly use the final score at the current step G(b_{m-1}^t) but take the product of all steps observed so far? If the constraints are satisfied at step t, doesn't that mean it's always satisfied at steps <t?\n\n\nPresentation Issues:\n1. The length reward function R (Eq 10) should be in the main paper.\n2. Section 2, \"Note that both models used in this paper are fine-tuned and parameter-fixed.\" what does this mean? Is it fixed or finetuned?\n3. Line 150, \"its global score at the termination is the maximum 1,\" Do you mean g_i?\n\nSuggestions:\n1. While the appendix shows some decoding examples, I think it would be helpful to visualize how this additional loss affects the search process by visualizing which constraints are \"active\" and which beam hypotheses are dropped due to incurring penalties on those constraints.\n\nOverall, this is an interesting work generalizing widely-used attention coverage losses. However, the approach taken in this paper needs to add a predictor and I'm not convinced how robust the performance is with respect to extra hyperparameters. Besides, I have some doubts about some results. Therefore, I'm not recommending its acceptance in its current form.\n\n==== Post Rebuttal ====\n\nOverall, some of my concerns have been addressed, but there seems to be some misunderstanding about previous works (it's equally possible that I might have misunderstood how attention coverage works). Therefore, I'm keeping my score for now and would appreciate it if the authors could post a short clarification.\n\n==== Post Post Rebuttal ====\n\nAfter discussing with the authors, I found that while bottom-up summarization uses stepwise coverage penalty, GNMT might not have used stepwise coverage. Therefore, my concern about wrong baseline implementation was not well founded, and I'm increasing my score to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper seeks to improve beam search for neural abstractive summarization. It deals with the local optimality problem of the original beam search: at each local step, the assumption is that the optimal hypothesis is within a top-k kept by the beam, but this is often not true. To alleviate this problem, the authors present a global attention mechanism that keeps track of cumulative local attention on the source tokens during decoding while penalizing deviation from the predicted global attention. Experiments on 9 summarization datasets show consistent gains.

The reviewers found many strong points, including a good motivation, a model that is principled and quite novel, and strong empirical results. The main negative points are that the work was not applied to other generation tasks, but extending the approach to, e.g., MT would require several changes to the model (as explained in the author response) and new experiments that would probably not fit in a single paper. Some of the reviewers found some aspects of the presentation confusing, but this could be fixed in the camera-ready version (I agree with the reviewer who suggested adding running example). Related work is a bit short and could mention attempts to mitigate exposure bias (which is not a completely unrelated problem, and its relatedness came up in the discussion) and other ways to get around limitations caused by token-level objectives. 

Minor note: The paper makes it sound like beam search was invented in 2016, but it is fundamental algorithm of AI that goes back to much earlier than that. If they want to put a citation, the authors might be better off citing an AI textbook such as Russell & Norvig (or find its earliest use there). If the authors wanted to refer to the first use of beam search in a neural seq2seq setting, I think [Graves, 2012] would be a more appropriate citation (it is used with RNNs, but the core algorithm remains the same). Prior to seq2seq, beam search was used extensively in NLG, including machine translation [Koehn, 2004].

Koehn, 2004: http://homepages.inf.ed.ac.uk/pkoehn/publications/pharaoh-amta2004.pdf

Graves, 2012: https://arxiv.org/abs/1211.3711 
 
Decision: Accept (Poster)