Venue: NAACL 2021

# Attention Head Masking for Inference Time Content Selection

in Abstractive Summarization

Shuyang Cao

Lu Wang

Computer Science and Engineering

University of Michigan

Ann Arbor, MI

{caoshuy, wangluxy}@umich.edu

###### Abstract

How can we effectively inform content selection in Transformer-based abstractive summarization models? In this work, we present a simple-yet-effective _attention head masking_ technique, which is applied on encoder-decoder attentions to pinpoint salient content at inference time. Using attention head masking, we are able to reveal the relation between encoder-decoder attentions and content selection behaviors of summarization models. We then demonstrate its effectiveness on three document summarization datasets based on both _in-domain_ and _cross-domain_ settings. Importantly, our models outperform prior state-of-the-art models on CNN/Daily Mail and New York Times datasets. Moreover, our inference-time masking technique is also data-efficient, requiring less than 20% of the training samples to outperform BART fine-tuned on the full CNN/DailyMail dataset.

+
Footnote †: 1 Our code is available at: [https://shuyangcao.github.io/projects/inference_head_masking](https://shuyangcao.github.io/projects/inference_head_masking).

## 1 Introduction

Large pre-trained Transformers have achieved state-of-the-art results on various summarization datasets with a fine-tuning phase to streamline the summarization pipeline Lewis et al. (2020); Yan et al. (2020). Yet, it is still unclear _how one can use large models more effectively for abstractive summarization_ . For example, prior work shows that informing content selection via attention weight updating in recurrent neural networks can further boost summarizer performance Gehrmann et al. (2018). However, with multi-heads attentions at all layers in Transformers Vaswani et al. (2017), highlighting salient content becomes non-trivial.

In this work, we propose an **inference-time attention head masking mechanism** that works on encoder-decoder attentions to underscore salient content from the source and improve the quality of abstractive summaries. Based on this mechanism, we first demonstrate the relation between encoder-decoder attentions and content selection behaviors, on three summarization datasets of CNN/DailyMail (CNN/DM), New York Times (NYT), and XSum. Second, we study whether multiple heads at the same layer collectively guide the summarization. Partial masking is found to be most effective, indicating a strong collaborative effect and the importance of head selection.

Based on these observations, we evaluate attention head masking on summarization benchmarks with salience labels provided by externally trained content selectors. On all three datasets, our model consistently outperforms fine-tuned BART Lewis et al. (2020) and several top performing Transformer-based abstractive summarization models Zhang et al. (2019); Yan et al. (2020). Summaries generated by our model are also considered to have better informativeness by human judges. Moreover, we illustrate that attention head masking is **data-efficient**: on CNN/DM, BART fine-tuned on less than \(20\%\) of the training data outperforms a version trained on the full set. Finally, we show that our method is effective under a **cross-domain** setting. With a content selector trained on NYT, BART fine-tuned on CNN/DM gains more than three points of ROUGE scores when tested on NYT articles.1

Footnote 1: Our code is available at: [https://shuyangcao.github.io/projects/inference_head_masking](https://shuyangcao.github.io/projects/inference_head_masking).

## 2 Related Work

**Large Pre-trained Models for Summarization.** Many recent advancements in text summarization have been achieved by large pre-trained language models Zhang et al. (2019); Liu and Lapata (2019); Song et al. (2019); Zhang et al. (2019). In particular, BART has demonstrated impressive performance on summarization, and is used as the base model in this work. Nonetheless, all prior attempts take pre-trained models as is and conduct fine-tuning on target datasets, without knowing if

it is the most effective usage. In contrast, we bring insights into the relation between attentions and content selection via masking operations to further improve summarization performance.

**Content Selection for Abstractive Summarization.** Content selection is a crucial step, where salient information is first detected and then summarized into concise abstracts (Chen and Bansal, 2018; Xu and Durrett, 2019). To minimize the propagation of selection errors, content selection is modeled as an extra component and learned within an end-to-end trained model (Zhou et al., 2017; Li et al., 2018; Gehrmann et al., 2018). To the best of our knowledge, we are the first to apply masks on selected layers and attention heads in Transformers for content selection in summarization. Moreover, our masking mechanism is only activated during inference, without any model modification.

**Analyzing Multi-head Attentions** has attracted growing interests in the NLP community (Clark et al., 2019; Kovaleva et al., 2019). Among the work that is relevant to encoder-decoder attentions, Michel et al. (2019) and Voita et al. (2019) observe that only a small portion of heads is relevant for translation and encoder-decoder attentions tend to be more important than self-attentions. Meanwhile, word alignments for machine translation are induced from encoder-decoder attention weights (Li et al., 2019; Kobayashi et al., 2020). However, none of prior work employs attentions to improve generation quality. As far as we are aware, this is the first work that studies the content selection effects of encoder-decoder attentions and uses them to guide better summary generation.

## 3 Attention Head Masking

We adopt large pre-trained sequence-to-sequence Transformer models (BART, specifically) for abstractive summarization. Transformer is built with **multi-head attentions**. Attentions are computed per step based on a query \({\bf q}\) along with the key and value matrices, \({\bf K}\) and \({\bf V}\):

\[{\rm Attention}({\bf q},{\bf K},{\bf V})={\rm softmax}(\frac{{\bf q}{\bf K}^ {T}}{\sqrt{d_{k}}}+{\bf m}){\bf V} \tag{1}\]

where \(d_{k}\) is a scaling factor and \({\bf m}\) is for padding or masking future tokens (when the value is \(-\infty\)).

**Masking Operation.** We propose attention head masking in encoder-decoder attentions, which blocks attentions to unimportant tokens, to better concentrate multi-head attentions on salient input tokens. Importantly, it is activated during inference. Concretely, we add an \(\tilde{\bf m}\) inside the softmax operator of Eq. 1, with implementation displayed in Fig. 1. The size of \(\tilde{\bf m}\) is the same as the input length. If the \(i\)-th token is tagged as salient, the corresponding element in \(\tilde{\bf m}\) is set to \(0\) (attendable to the attention heads), and \(-\infty\) otherwise (hidden from these heads). The saliency labels can be predicted by an externally trained content selector.

## 4 Encoder-decoder Attentions and Content Selection

In this section, we first probe into the content selection behavior of each single head (SS 4.1), and then study the synergism among heads at the same layer (SS 4.2). In SS 4.3, we analyze the attentions' focus.

Our analysis is conducted on CNN/DM (Hermann et al., 2015), NYT (Consortium and Company, 2008), and XSum (Narayan et al., 2018). We follow Lewis et al. (2020) for data preprocessing and train/validation/test splits on CNN/DM and XSum, and adopt the setups in Paulus et al. (2018) for NYT, except that we keep entities and numbers. The number of samples in training, validation, and test set are: \(287{,}188\), \(13{,}367\) and \(11{,}490\) for CNN/DM; \(588{,}909\), \(32{,}716\) and \(32{,}703\) for NYT; \(204{,}045\), \(11{,}332\) and \(11{,}334\) for XSum.

For experiments in this section, we create an _analysis set_ of \(1{,}000\) random samples from the validation split of each dataset to reduce computational cost.

### Content Selection Effects

First, we study the feasibility of using encoder-decoder attentions to inform content selection and subsequently boost summary informativeness. Concretely, we apply attention head masking based on oracle content selection labels (henceforth **or

Figure 1: Illustration of attention head masking (\(\tilde{\bf m}\)).

 

acle masking**). Oracle labels are constructed by aligning a reference summary to the source article, where we iteratively find the longest common subsequences between the two.

Taking a fine-tuned BART model, we apply oracle masking on each head at each layer when decoding on the analysis set. The ROUGE score obtained in this setting is denoted as \(r_{ora}\). We then apply uniform encoder-decoder attention weights over the source to build a baseline that mimics no content selection, inspired by Wiegreffe and Pinter (2019). This yields a ROUGE score of \(r_{uni}\). The **content select effect** per head can thus be calculated as the ROUGE improvement, i.e., \(r_{ora}-r_{uni}\).

Overall, it is more effective to constrain attentions to salient content at the top layers, according to the results on CNN/DM in Fig. 2. Specifically, with oracle masking, the top layer yields the most ROUGE-1 improvement. We observe similar trends on NYT and XSum (figures are in Appendix C). This indicates the feasibility of **leveraging attention head masking to improve summary informativeness.**

### Synergism Analysis

Next, we study whether masking multiple heads can further boost content selection and whether they form synergy. On the left of Fig. 3, we show content selection effect by gradually applying oracle masking on more heads at each layer, with heads sorted based on individual ROUGE improvements. Notably, the most ROUGE-1 improvement is achieved by masking 15 (out of 16) heads at the top layer, suggesting **a strong collaborative effect on content selection by masking multiple heads**.

We further compare the ROUGE score gains between oracle masking on all heads and the sum of individual effects, illustrated on the right of Fig. 3. The discrepancies between the two values suggest that the heads may not be independent at pinpointing salient content. In Appendix D, we reach similar results on NYT and XSum.

Based on the above observations, we argue that **it is necessary to select layers and heads accordingly to achieve the best content selection effect**, with more summarization results reported in SS 5.

### Attention Focus

We further provide a fine-grained study on what types of words the heads attend to. Concretely, we consider each word generated during decoding, denoted as \(y\). Given an attention head, we follow the highest attention weight to identify the input word \(x\) ("**attendee**"). We study several categories of attendee \(x\): (1) word in the reference (salient); (2) content word; (3) the first and last words in the document. For salient and content, we further consider two subcategories: \(x=y\) (copy) and \(x\neq y\) (non-copy). We then tally the occurrences of each type of attendees per head at each layer on the analysis set.

We show the percentages of copy and non-copy salient attendees, copy content attendees, and first attendees on CNN/DM in Fig. 4. As can be seen, top layers tend to focus on input tokens that will be generated as is, while bottom layers attend to salient words that are not used for current generation. Additionally, bottom layers fre

Figure 3: [**Left**] ROUGE-1 F1 improvement by incrementally applying oracle masking to the next head with most ROUGE improvement per layer on CNN/DM. Dotted lines indicate that the newly masked heads do not have individual ROUGE improvements. [**Right**] ROUGE-1 recall improvement by masking all heads vs. sum of improvement by masking each head separately on CNN/DM. Best viewed in color.

Figure 2: ROUGE-1 F1 improvement with oracle masks for each head at each layer on the analysis set of CNN/DM. Overall, top layers see greater improvement than bottom layers. Layer 1 is the bottom layer connected with the word embeddings.

 

quently attend to the first token of the document, where bottom layers are more likely performing context gathering. On NYT and XSum (figures are in Appendix E), similar trends are observed except that the first attendees are more focused by the top layers on NYT articles, where many of them start with all capitalized words.

## 5 Summarization Results with Attention Head Masking

In this section, we show how to leverage attention head masking and a content selector to improve summary informativeness on three datasets. We first train a binary sequence tagger for each dataset to label salient tokens in the source, used for **system masking** for attention heads. Our sequence tagger is a RoBERTa Liu et al. (2019) encoder followed by a double layer multilayer perceptron (MLP) with a hyperbolic tangent activation function in between. To obtain the probability for each token, the MLP output is further fed into a sigmoid activation function. Details for training and decoding are in Appendix A.

The decision boundary for the sequence tagger is selected according to the F1 score calculated between the predicted tags and the ground-truth labels on the validation set. We search for the best decision boundary from \(0.1\) to \(0.4\), with a step size of \(0.01\). The final decision boundaries used for taggers trained on CNN/DM, NYT, XSum are \(0.20\), \(0.24\), and \(0.18\), achieving ROUGE-1 F1 of \(43.70\), \(44.10\), and \(31.56\), respectively.

To select which heads at which layers to mask, we employ a greedy selection strategy. On the analysis set, we gradually apply system masking on four heads with most ROUGE improvement according to the study in SS 4.1, and we select the heads that achieve the highest sum of ROUGE-1 F1 and ROUGE-2 F1. We apply four heads each time to reduce computational cost of hyperparameter searching. Heads selected for each dataset are in Appendix B.

**In-domain Results.** Table 1 shows that applying our attention head masking technique on BART obtains significantly better results on CNN/DM and NYT, compared to several top performing abstractive summarization models trained with large Transformers. The improvement is more pronounced for CNN/DM than the other two datasets. We believe this is due to the difference in abstractiveness among the three datasets. CNN/DM has more extractive summaries compared to the other datasets Grusky et al. (2018), suggesting attention head masking is more effective on extractive datasets. Notably, PEGASUS is pre-trained with 3.8TB of news articles, the BART model used in our work is only pre-trained with 160GB of a com

Figure 4: copy and non-copy salient attendee word percentages on the analysis set of CNN/DM. Top layers focus on words to be “copied”, while bottom layers attend to the broader salient context.

 

bination of news, books, stories, and web text. The large size of the pre-training data might be a big contributor to the better performance by PEGASUS on XSum.

For **human evaluation**, we hire three fluent English speakers to rate \(50\) pairs of summaries generated with and without attention head masking based on BART for **informativeness** and **faithfulness**. Informativeness measures how well the summary captures salient content from the article, while faithfulness indicates whether the summary correctly reflects the content in the source article. The annotators are asked to determine if attention head masking improves any of the two aspects. As shown in Table 2 where all ratings by three judges are considered, summaries generated with attention head masking are considered to have better informativeness, but no substantial improvement on faithfulness is observed.

Limited Training DataNext, we study if our masking technique is still effective if given limited training samples. We use the limited training samples to train both the summarizer and the content selector. As can be seen in Fig. 5, our masking technique consistently increases ROUGE scores with varying amounts of training data. Notably, our model trained on only \(30\)K samples (with attention head masking) outperforms the model trained on the full dataset, suggesting that directly informing content selection is more data-efficient than model fine-tuning on more summaries.

Cross-domain ResultsFinally, we show results on NYT using BART fine-tuned on CNN/DM, with system masks predicted by a tagger trained on different sizes of NYT samples (Table 3). Using a selector trained with only \(10\)k of target domain samples, we already significantly improve the performance by BART trained on CNN/DM only.

## 6 Conclusion

We propose attention head masking that constrains encoder-decoder attention heads to attend to salient tokens, to inform content selection in abstractive summarization. With this technique, we first demonstrate the relation between encoder-decoder attentions and content selection behaviors. With system masks predicted by external content selectors, we show that attention head masking can consistently improve ROUGE scores over competitive summarization models on three benchmarks. Summaries generated with attention head masking are also preferred by human judges more frequently. Additional experiments demonstrate that our method is more data-efficient and effective on both in-domain and cross-domain settings.

### Acknowledgements

This research is supported in part by National Science Foundation through Grant IIS-1813341, and by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract # FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. We thank three anonymous reviewers for their constructive suggestions.

Figure 5: Results on CNN/DM with different sizes of training data. Our method consistently improves the summarizer.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **w/ masking** & **w/o masking** & **Tie** \\ \cline{2-4}
**Informativeness** & **36.0\%** & 19.3\% & 44.7\% \\
**Faithfulness** & **10.0\%** & 7.3\% & 82.7\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Percentages of summaries with and without attention head masking favored by annotators on informativeness and faithfulness. The Krippendorff’s \(\alpha\) for informativeness and faithfulness are \(0.30\) and \(0.47\).

 



## References

* C. Celikyilmaz, A. Bosselut, X. He, and Y. Choi (2018)Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 1662-1675. External Links: Link Cited by: SS3.4.
* Y. Chen and M. Bansal (2018)Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 675-686. External Links: Link Cited by: SS3.4.
* K. Clark, U. Khandelwal, O. Levy, and C. D. Manning (2019)What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Florence, Italy, pp. 276-286. External Links: Link Cited by: SS3.4.
* L. Data Consortium and N. York Times Company (2008)The new york times annotated corpus. Note: [http://www.acl.com/en/article/pdf/LLC/corpora](http://www.acl.com/en/article/pdf/LLC/corpora) External Links: Link Cited by: SS3.4.
* L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H. Hon (2019)Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32, Online, pp. 13063-13075. External Links: Link Cited by: SS3.4.
* S. Gehrmann, Y. Deng, and A. Rush (2018)Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 4098-4109. External Links: Link Cited by: SS3.4.
* M. Grusky, M. Naaman, and Y. Artzi (2018)Newsroom: a dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 708-719. External Links: Link Cited by: SS3.4.
* K. Moritz Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom (2015)Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, Online, pp. 1693-1701. External Links: Link Cited by: SS3.4.
* L. Huang, L. Wu, and L. Wang (2020)Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. External Links: Link Cited by: SS3.4.
* D. P. Kingma and J. Ba (2015)Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Cited by: SS3.4.
* G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui (2020)Attention is not only a weight: analyzing transformers with vector norms. External Links: Link Cited by: SS3.4.
* O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky (2019)Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 4365-4374. External Links: Link Cited by: SS3.4.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7871-7880. External Links: Link Cited by: SS3.4.
* W. Li, X. Xiao, Y. Lyu, and Y. Wang (2018)Improving neural abstractive document summarization with explicit information selection modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 1787-1796. External Links: Link Cited by: SS3.4.
* X. Li, G. Li, L. Liu, M. Meng, and Shuming Shi (2019)On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 1293-1303. External Links: Link Cited by: SS3.4.
* Y. Liu and M. Lapata (2019)Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3730-3740. External Links: Link Cited by: SS3.4.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. External Links: Link Cited by: SS3.4.
* P. Michel, O. Levy, and G. Neubig (2019)Are sixteen heads really better than one?. In Advances in Neural Information Processing Systems

volume 32, pp. 14014-14024. Curran Associates, Inc. Cited by: SS5.
* S. Narayan, S. B. Cohen, and M. Lapata (2018)Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 1797-1807. External Links: Link, Document Cited by: SS6.
* M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli (2019)fairseq: a fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, Cited by: SS1, SS2.
* R. Paulus, C. Xiong, and R. Socher (2018)A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, Cited by: SS1, SS2.
* K. Song, X. Tan, T. Qin, J. Lu, and Y. Liu (2019)MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, Online, pp. 97 of _Proceedings of Machine Learning Research_, Long Beach, California, USA, PMLR. External Links: ISBN 978-1-4503-3213-X, Link, Document Cited by: SS1, SS2.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30, pp. 5998-6008. External Links: Link, Document Cited by: SS1, SS2.
* E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov (2019)Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 5797-5808. External Links: Link, Document Cited by: SS1, SS2.
* S. Wiegreffe and Yuval Pinter (2019)Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 11-20. External Links: Link, Document Cited by: SS1, SS2.
* J. Xu and G. Durrett (2019)Neural extractive text summarization with syntactic compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3292-3303. External Links: Link, Document Cited by: SS1, SS2.
* Y. Yan, W. Qi, Y. Gong, D. Liu, N. Duan, J. Chen, R. Zhang, and M. Zhou (2020)Prophetnet: predicting future n-gram for sequence-to-sequence pre-training. External Links: 2009.00226 Cited by: SS1, SS2.
* H. Zhang, J. Cai, J. Xu, and J. Wang (2019)Pretraining-based natural language generation for text summarization. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), Hong Kong, China, pp. 789-797. External Links: Link, Document Cited by: SS1, SS2.
* J. Zhang, Y. Zhao, M. Saleh, and P. J. Liu (2019)Pegasus: pre-training with extracted gap-sentences for abstractive summarization. External Links: 1905.00226 Cited by: SS1, SS2.
* Q. Zhou, N. Yang, F. Wei, and M. Zhou (2017)Selective encoding for abstractive sentence summarization. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). External Links: Link, Document Cited by: SS1, SS2.

## Appendix A Training and Decoding Settings

When training the sequence taggers, we minimize the average binary cross-entropy of each token's selection probability relative to the ground-truth label. The parameters of the RoBERTa encoder are fixed. We set the learning rate to \(5\times 10^{-4}\) and batch size to \(128\). Unless specified, all the models in this paper are trained with Adam Kingma and Ba (2015) optimizer and training will be stopped if there is no improvement on the validation set for \(2\) consecutive epochs. For BERT models, we follow the instructions provided by Fairseq Ott et al. (2019) to set the training hyperparameters on CNN/DM and XSum. We use the same hyperparameters for CNN/DM and NYT, except that we adopt a linear learning rate decay of \(30{,}000\) steps in total for NYT. During test, we use a beam size of \(5\), \(5\), \(6\) for CNN/DM, NYT, and XSum, respectively. To reduce computational cost, we use beam size \(1\) for our analysis experiments on all datasets. The length penalties are \(2.0\), \(1.5\) and \(1.0\) for CNN/DM, NYT, and XSum, following Lewis et al. (2020). We set the minimal and maximal lengths during decoding as: \(55\) and \(140\) for CNN/DM, \(0\) and \(140\) for NYT, and \(10\) and \(60\) for XSum.

## Appendix B Head Selection

For CNN/DM, we apply masking to all heads at layer 1. The ROUGE-1/2/L F1 on the analysis set are \(36.43/16.02/33.59\). For NYT, we apply masking to 12 heads at layer 3. The indices of heads are: 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. The ROUGE-1/2/L F1 on the analysis set are \(55.27/39.20/48.16\). For XSum, we apply masking to 12 heads at layer 3. The indices of heads are: 1, 2, 3, 4, 6, 7, 

8, 9, 11, 13, 14, 15. The ROUGE-1/2/L F1 on the analysis set are \(45.77/22.82/37.60\).

## Appendix C Content Selection Effects on XSum and NYT

The content selection effects for BART models fine-tuned on XSum and NYT, measured by the ROUGE improvement from the uniform attention weight setting to the oracle masking setting, are shown in Fig. 6.

On all three datasets, it is more effective to constrain attentions to salient content at the top layers. Especially, the top layer yields the most ROUGE-1 improvement. Moreover, the ROUGE improvement by a specific head varies among different datasets.

## Appendix D Additional Results for Synergism Analysis

We show the synergism analysis for models finetuned on XSum and NYT in Fig. 7. They both echo the observation on CNN/DM that multiple heads have strong collaborative effects and heads may not be independent at pinpointing different salient content.

## Appendix E Attention Focus

We show the percentages of each type of attendees on the analysis sets of XSum, NYT, and CNN/DM in Fig. 8, Fig. 9, and Fig. 10, respectively. We find that heads have similar focus for salient words, content words, and the last word across different datasets. Interestingly, the attention focus for the first word on NYT is different from other datasets. On NYT, many articles start with all capitalized words, which might become the focus of some heads.

Figure 6: ROUGE-1 improvement with oracle masks for each head at each layer on the analysis sets of XSum and NYT.

Figure 7: **[Left]** ROUGE-1 F1 improvement by incrementally applying oracle masking to the next head with most ROUGE improvement per layer on XSum and NYT. Dotted lines indicate that the newly masked heads do not have individual ROUGE improvements. **[Right]** ROUGE-1 recall improvement by masking all heads vs. sum of improvement by masking each head separately on XSum and NYT. Better displayed with color.

 

Figure 8: Percentages of copy salient, non-copy salient, copy content, non-copy content, first and last attendees for each head at each layer on the analysis set of XSum.

Figure 10: Percentages of non-copy content and last attendees for each head at each layer on the analysis set of CNN/DM.

Figure 9: Percentages of copy salient, non-copy salient, copy content, non-copy content, first and last attendees for each head at each layer on the analysis set of NYT.