SlowGuess/NAACL_2021_OCR huggingface dataset:
Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. 2018. Improving neural abstractive document summarization with explicit information selection modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1787-1796, Brussels, Belgium. Association for Computational Linguistics.
Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1293-1303, Florence, Italy. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems,
volume 32, pages 14014-14024. Curran Associates, Inc.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936, Long Beach, California, USA. PMLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3292-3303, Hong Kong, China. Association for Computational Linguistics.
Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training.
Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019a. Pretraining-based natural language generation for text summarization. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 789-797, Hong Kong, China. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019b. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective encoding for abstractive sentence summarization. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
A Training and Decoding Settings
When training the sequence taggers, we minimize the average binary cross-entropy of each token's selection probability relative to the ground-truth label. The parameters of the RoBERTa encoder are fixed. We set the learning rate to  $5 \times 10^{-4}$  and batch size to 128. Unless specified, all the models in this paper are trained with Adam (Kingma and Ba, 2015) optimizer and training will be stopped if there is no improvement on the validation set for 2 consecutive epochs.
For BART models, we follow the instructions provided by Fairseq (Ott et al., 2019) to set the training hyperparameters on CNN/DM and XSum. We use the same hyperparameters for CNN/DM and NYT, except that we adopt a linear learning rate decay of 30,000 steps in total for NYT.
During test, we use a beam size of 5, 5, 6 for CNN/DM, NYT, and XSum, respectively. To reduce computational cost, we use beam size 1 for our analysis experiments on all datasets. The length penalties are 2.0, 1.5 and 1.0 for CNN/DM, NYT, and XSum, following Lewis et al. (2020). We set the minimal and maximal lengths during decoding as: 55 and 140 for CNN/DM, 0 and 140 for NYT, and 10 and 60 for XSum.
B Head Selection
For CNN/DM, we apply masking to all heads at layer 1. The ROUGE-1/2/L F1 on the analysis set are 36.43/16.02/33.59.
For NYT, we apply masking to 12 heads at layer 3. The indices of heads are: 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. The ROUGE-1/2/L F1 on the analysis set are  $55.27 / 39.20 / 48.16$ .
For XSum, we apply masking to 12 heads at layer 3. The indices of heads are: 1, 2, 3, 4, 6, 7,
(a) XSum
(b) NYT
Figure 7: [Left] ROUGE-1 F1 improvement by incrementally applying oracle masking to the next head with most ROUGE improvement per layer on XSum and NYT. Dotted lines indicate that the newly masked heads do not have individual ROUGE improvements. [Right] ROUGE-1 recall improvement by masking all heads vs. sum of improvement by masking each head separately on XSum and NYT. Better displayed with color.
8, 9, 11, 13, 14, 15. The ROUGE-1/2/L F1 on the analysis set are  $45.77 / 22.82 / 37.60$ .
C Content Selection Effects on XSum and NYT
The content selection effects for BART models fine-tuned on XSum and NYT, measured by the ROUGE improvement from the uniform attention weight setting to the oracle masking setting, are shown in Fig. 6.
On all three datasets, it is more effective to constrain attentions to salient content at the top layers. Especially, the top layer yields the most ROUGE-1 improvement. Moreover, the ROUGE improvement by a specific head varies among different datasets.
D Additional Results for Synergism Analysis
We show the synergism analysis for models finetuned on XSum and NYT in Fig. 7. They both echo the observation on CNN/DM that multiple heads
Figure 6: ROUGE-1 improvement with oracle masks for each head at each layer on the analysis sets of XSum and NYT.
(b) NYT
have strong collaborative effects and heads may not be independent at pinpointing different salient content.
E Attention Focus
We show the percentages of each type of attendees on the analysis sets of XSum, NYT, and CNN/DM in Fig. 8, Fig. 9, and Fig. 10, respectively. We find that heads have similar focus for salient words, content words, and the last word across different datasets. Interestingly, the attention focus for the first word on NYT is different from other datasets. On NYT, many articles start with all capitalized words, which might become the focus of some heads.
(b) NON-COPY SALIENT
(c) COPY CONTENT
(e) FIRST
(d) NON-COPY CONTENT
(f) LAST
(a) NON-COPY CONTENT
(b) LAST
Figure 10: Percentages of NON-COPY CONTENT and LAST attendees for each head at each layer on the analysis set of CNN/DM.
Figure 8: Percentages of COPY SALIENT, NON-COPY SALIENT, COPY CONTENT, NON-COPY CONTENT, FIRST and LAST attendees for each head at each layer on the analysis set of XSum.
(c) COPY CONTENT
(e) FIRST
Figure 9: Percentages of COPY SALIENT, NON-COPY SALIENT, COPY CONTENT, NON-COPY CONTENT, FIRST and LAST attendees for each head at each layer on the analysis set of NYT.