Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes to add a global attention feature component to the beam search scoring procedure for conditional generation, which typically just considers the fitness of successor tokens conditioned on the predicted history of tokens in the beam. This augmentation basically keeps track of cumulative local autoregressive attention on the source tokens during beam search and penalizes major deviation from the predicted desired "global" attention over the source tokens. Specifically, this approach proposes a finetuning step of neural language generators, which is responsible for predicting cumulative global attention on each token and the training is done via regression. During training, the "reference" global attention is computed by using the full reference in the decoder transformer instead of just the left prefix of the reference to compute attention.

A relatively simple scheme is proposed to measure deviation of cumulative local attention from the global attention in a decomposable iterative manner. This scheme also involves a form of length reward that encourages the generation to match the length predicted by the global attention.

Experiments are performed on the task fo summarization with pretrained summarization system on 9 datasets. The results show that the proposed beam search augmentation outperforms standard beam search in general. Most strikingly, the ablation experiment with the "Oracle" global attention (instead of prediction, if we had access to the actual global attention and hence the actual reference length) significantly outperforms beam search and the proposed approach demonstrating the potential of global attention.
main_review: Positives:

-- The proposed approach is well motivated, very simple to implement and novel to my knowledge.

-- The results on abstractive summarization show that the proposed approach is effective. Specifically, the oracle experiments clearly show that global attention, although not used during training of the seq2seq models, is a useful signal to model.

Negatives and Questions:

-- The experiments are only on the task of summarization where the token overlap is huge. I would have liked to see the performance of this approach on other related tasks like machine translation.

-- It took me several readings to figure out what exactly was trained/finetuned--and I am not completely sure even now. I believe that the seq2seq models were pretrained and used local attention as is standard during training. The architectures were finetuned to predict global attention. While it is clear that finetuned models were used for global attention prediction, were they also used for decoding or were the pretrained models used for performing the beam search.

-- It is surprising that global attention would be useful given that it is not at all used during training of seq2seq models. Is it mainly useful because of token overlap between source and reference--i.e. while training global attention can use reference to tell the decoder which source tokens are important for the future steps? Again, this is a question that would be better answered by experiments on another task like MT.

-- On the second point, while the proposed approach is compared to beam search, I think there are two beam search baselines to compare to. 1) Beam search with the pretrained BART/PEGASUS, 2) Beam search with finetuned BART/PEGASUS.

-- Also, what happens if the seq2seq training includes global attention (or the full seq2seq training is finetuned with global attention)? This would mean that in addition to finetuning with the regression loss, you would perform finetuning with the actual seq2seq cross entropy loss with global attention.
limitations_and_societal_impact: Yes, I found the sparse discussion sufficient given the mostly algorithmic contribution of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tackles the problem of suboptimality of beam search as it is not able to have a global perspective of the probability of the generated beams due to truncated beam size (less than vocabulary size) for space and time complexity purposes. This paper proposes a global-aware beam search which is calibrated using the global attention distribution to guide the local decisions at every time step during beam search. The global attention distribution prediction is formulated as a regression task, which is then fed into a global scoring function to guide the beam search-based generation. This approach solves the mismatch between the local optimality of beam search with the global optimal generation. This modified search strategy has been shown to boost the summarization of two SOTA models BART and PEGASUS.

main_review: ## Strong points:
1. The paper is well-written and clear.
2. The problem is well-motivated and theoretically sound.
3. The idea of using attention distribution to guide beam search in a principled manner is novel and interesting.
4. The experimental result improvements are convincing across all 9 summarization datasets.
5. Thorough analysis has been done to compare with beam search across 9 summarization datasets, in addition to comparison with other methods like coverage, repetition penalty, attention masking, along with ablation studies across various dimensions.
6. The proposed algorithm has also been shown to be robust by using attention prediction from a different dataset.


## Questions:
1. It is not clear (or maybe there is some gap in my understanding), whether the predicted attention distribution still leads to the same theoretical result (Theorem 3.1 and Corollary 3.1.1)
2. It will be interesting to see how this decoding strategy fares in other NLG text2text tasks like Paraphrasing, Neural Machine Translation, etc.
3. How does the proposed strategy compare with other decoding strategies like Top-k Sampling[1], Nucleus Sampling[2]?
4. For completeness, human evaluation with respect to correctness, adequacy, fluency, and coherence will be a helpful comparison.
5. The related work can include more guided decoding strategies as summarized in the blog[3].

## Originality: 7/10.

## Quality: 8/10.

## Clarity: 8/10.

## Significance: 7/10.

REFERENCES:
[1] Hierarchical Neural Story Generation, Fan et al.
[2] The Curious Case of Neural Text Degeneration, Holtzman et al.
[3] Controllable Neural Text Generation: https://lilianweng.github.io/lil-log/2021/01/02/controllable-neural-text-generation.html


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method to improve beam search by learning and predicting the global attention. A global scoring function is developed to regulate beam search to generate summaries. In Phase I, the global attention distribution is predicted in order to be included as a protocol to calibrate beam search. In Phase II, a step-wise global scoring function is used to guide beam search based on the predicted global attention distribution. 
Experiments on 9 datasets show that the global-aware inference improves state-of-the-art summarization models. 

main_review: The angle and direction is pretty interesting. While most of prior work on improving beam search focuses on repetition, similarity/dissimilarity, and length control, this paper presents a relatively new angle. The method proposed can implicitly improve the content planning procedure of text summarization and generation. 

The performance improvement is significant. Comparing with other methods (Table 4), it is also competitive. 

Question:
Line 96 - 98: “With both sources and  reference targets available in the training set, the attention given by reference tokens is computable, called global cross attention αg ”			 				
			
		


limitations_and_societal_impact: Presentation 
Although I believe the idea of this idea is actually pretty straightforward, the presentation and illustration of the idea are hard to follow. There are just so many notations, superscripts and subscripts, like alpha {l,k,t,i}, zeta, G, etc. I believe the presentation can be significantly improved. I would like to see a figure with a toy running example, which I think will walk through how the method works, during training and during inference. Something like source = A B C, decoding prefix = y0 y1, the local attention distribution is [0.1, 0.2, 0.7], the global attention is [....],try to minimize the distance of __ and __, during inference, step 0, step 1, etc. 
If the authors want to convince more people to use this method, this step is necessary, from my point of view. 
The naming and symbols could be simplified like “local cross attention as the dynamic local attention distribution”.

The method is a little hacky and complicated, and I would like to see more analysis about how attention changes, what’s the global and local attention. The evaluation mostly focuses on ROUGE, but I still don’t have much idea about what happened and what’s the change of the attention pattern or behavior. If this piece can be shown, I believe the paper will seem less hacky because you explain how it works.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a generalization of the attention coverage penalty in seq2seq models (Wu et al 2016), where instead of encouraging a uniform coverage of the source, this work first predicts the expected coverage of each source word. During beam search, beam hypotheses that violate the expected attention coverage constraints receive a penalty. Experiments on 9 summarization benchmarks show that the proposed coverage loss outperforms using existing attention coverage losses during beam search.
main_review: Strengths:
1. This work generalizes the attention penalty loss, where different variants are used in different applications (for example, translation often encourages uniform coverage of the source, whereas summarization often penalizes repeatedly attending to the same word to reduce repetitions). In theory, the expected attention coverage predictor can learn to vary its behavior according to the task, instead of relying on the expert design of the form of coverage loss.
2. The approach proposed by this paper can be viewed as maintaining similar statistics between training and testing (similar to professor forcing), and this might inspire future research matching other statistics such as the distribution of hidden states.
3. Experiments show improvements over existing coverage mechanisms.

Weaknesses:
1. In Table 4, the last row shows that Global Aware has R1, R2, RL of 45.57, 22.60, 37.61, but according to table 3, this is the performance of Global Aware without length reward, whereas the performance of Global Aware in table 3 is 45.44, 22.15, 37.11.
2. While the proposed approach might alleviate the locality bias of beam search, it still suffers from a similar locality bias. The attention constraints (Eq 7) only become active when the accumulated attention exceeds its total budget, which I expect to happen rarely at the beginning of the search (when accumulated attentions are small) and more frequently towards the end of the search (when accumulated attentions are larger). Therefore, I wouldn't expect this "global term" to guide early decisions during search, and this is exactly where beam search has lots of uncertainty and global planning is needed the most. For this reason, I don't think the proposed approach (or existing attention coverage penalties) is truly "global aware" unless there's some estimate of future rewards (https://arxiv.org/pdf/1701.06549.pdf).
3. In practice, the proposed approach introduces additional components: an expected total attention predictor, as well as additional hyperparameters. While using empirical hyperparameters for some settings results in good performances, I'm not totally convinced how robust the performance is with respect to hyperparameters, and how much tuning effort is needed. 
4. The comparison to the baseline coverage in table 4 does not appear fair. According to my understanding coverage doesn't include length penalty, and if we compare baselines in table 4 with Global Aware without length reward in table 3, coverage underperforms on XSUM but outperforms on CNN/DM, and the other baselines also appear to have similar performances to the proposed method.
5. In Figure 2, the performance changes are not significant, and I think using even larger beam sizes would show a clearer trend.

Questions for authors:
1. How well does your expected attention predictor predict attentions? Is it predictable just from the source? Wouldn't different summaries for the same document pick source words differently?
2. In Eq 8 why not directly use the final score at the current step G(b_{m-1}^t) but take the product of all steps observed so far? If the constraints are satisfied at step t, doesn't that mean it's always satisfied at steps <t?


Presentation Issues:
1. The length reward function R (Eq 10) should be in the main paper.
2. Section 2, "Note that both models used in this paper are fine-tuned and parameter-fixed." what does this mean? Is it fixed or finetuned?
3. Line 150, "its global score at the termination is the maximum 1," Do you mean g_i?

Suggestions:
1. While the appendix shows some decoding examples, I think it would be helpful to visualize how this additional loss affects the search process by visualizing which constraints are "active" and which beam hypotheses are dropped due to incurring penalties on those constraints.

Overall, this is an interesting work generalizing widely-used attention coverage losses. However, the approach taken in this paper needs to add a predictor and I'm not convinced how robust the performance is with respect to extra hyperparameters. Besides, I have some doubts about some results. Therefore, I'm not recommending its acceptance in its current form.

==== Post Rebuttal ====

Overall, some of my concerns have been addressed, but there seems to be some misunderstanding about previous works (it's equally possible that I might have misunderstood how attention coverage works). Therefore, I'm keeping my score for now and would appreciate it if the authors could post a short clarification.

==== Post Post Rebuttal ====

After discussing with the authors, I found that while bottom-up summarization uses stepwise coverage penalty, GNMT might not have used stepwise coverage. Therefore, my concern about wrong baseline implementation was not well founded, and I'm increasing my score to 6.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper thinks that cross attention distribution in transformer’s encoder-decoder framework can help the performance of beam search. At the same time, this paper also proposes an algorithm to predict distribution to help beam search. This paper conduct experiments on 9 datasets to prove its effectiveness.

main_review: There is no doubt that the empirical results given in this paper are quite strong, but it is difficult for me to understand every detail of the paper and the motivation of the paper.

Is there a possibility that the performance improvement is obtained by the increase of model parameters for attention prediction? Usually, better performance can be obtained by increasing the model parameters.
"""
197 Implementation Details. We adopt a randomly initialized 2-layer transformer-encoder in the
198 attention-prediction model, and the structure of each layer is the same as the BART-encoder layer.
"""

limitations_and_societal_impact: see review
review_title: null

Meta-review: The paper seeks to improve beam search for neural abstractive summarization. It deals with the local optimality problem of the original beam search: at each local step, the assumption is that the optimal hypothesis is within a top-k kept by the beam, but this is often not true. To alleviate this problem, the authors present a global attention mechanism that keeps track of cumulative local attention on the source tokens during decoding while penalizing deviation from the predicted global attention. Experiments on 9 summarization datasets show consistent gains.

The reviewers found many strong points, including a good motivation, a model that is principled and quite novel, and strong empirical results. The main negative points are that the work was not applied to other generation tasks, but extending the approach to, e.g., MT would require several changes to the model (as explained in the author response) and new experiments that would probably not fit in a single paper. Some of the reviewers found some aspects of the presentation confusing, but this could be fixed in the camera-ready version (I agree with the reviewer who suggested adding running example). Related work is a bit short and could mention attempts to mitigate exposure bias (which is not a completely unrelated problem, and its relatedness came up in the discussion) and other ways to get around limitations caused by token-level objectives. 

Minor note: The paper makes it sound like beam search was invented in 2016, but it is fundamental algorithm of AI that goes back to much earlier than that. If they want to put a citation, the authors might be better off citing an AI textbook such as Russell & Norvig (or find its earliest use there). If the authors wanted to refer to the first use of beam search in a neural seq2seq setting, I think [Graves, 2012] would be a more appropriate citation (it is used with RNNs, but the core algorithm remains the same). Prior to seq2seq, beam search was used extensively in NLG, including machine translation [Koehn, 2004].

Koehn, 2004: http://homepages.inf.ed.ac.uk/pkoehn/publications/pharaoh-amta2004.pdf

Graves, 2012: https://arxiv.org/abs/1211.3711 
 
Decision: Accept (Poster)