SlowGuess/NAACL_2021_OCR huggingface dataset:
Complementarity of Different Guidance Signals. While some guidance signals achieve worse performance than others, it is still possible to aggregate their outputs and obtain better performance if their outputs are diverse and they complement each other. To verify this hypothesis, we try to select the best output of the four guidance signals for each test datapoint and investigate if we can aggregate their best outputs and achieve better performance.
Concretely, for each test input, we perform an oracle experiment where we compute the ROUGE
Model	Reddit	XSum	WikiHow	PubMed	NYT
R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Oracle	36.21	13.74	28.93	29.79	8.81	22.66	35.59	12.98	32.68	45.12	20.33	40.19	58.44	38.39	50.00
Extractive
BertExt (Base)	23.86	5.85	19.11	22.86	4.48	17.16	30.40	8.67	28.32	40.29	14.37	35.88	45.98	25.29	42.46
MatchSum	25.09	6.17	20.13	24.86	4.66	18.41	31.85	8.98	29.58	41.21	14.91	36.75	46.98	26.67	43.62
Bert-Based
BertAbs	26.92	6.35	19.81	38.76	16.33	31.15	38.16	15.06	34.71	36.04	12.16	29.02	49.94	31.44	46.67
Ours (BertAbs + MatchSum)	26.89	6.75	20.35	38.77	16.14	30.96	38.29	15.10	34.80	37.82	12.32	30.53	50.50	31.57	47.24
BART-Based
BART	35.00	12.89	27.96	45.51	21.94	36.75	41.46	17.80	39.89	44.72	16.48	41.00	54.13	35.15	47.00
Ours (BART + MatchSum)	34.52	12.71	27.58	45.40	21.89	36.67	41.74	17.73	40.09	45.09	16.72	41.32	54.27	35.37	47.63
Table 5: Results of our model guided with highlighted sentences on five datasets. Highest numbers in each section are in bold. We use MatchSum to predict the guidance at test time. Extractive results are from Zhong et al. (2020).
Sentence	Win [%]	Combined
R-1/R-2/R-L
Keyword	Relation	Retrieve
39.28	19.55	21.12	20.05	48.30/25.25/45.15
Table 6: No guidance signals can outperform all the other ones for all the test data, and aggregating the best outputs of the four guided models achieves significant improvements over the best single guided model (43.78/20.66/40.66 R-1/R-2/R-L scores).
Sentence	Keyword	Relation	Retrieve
Sentence	43.78	46.11	46.20	46.27
Keyword	-	42.21	44.39	44.35
Relation	-	-	41.40	44.60
Retrieve	-	-	-	40.88
Table 7: Combining the best outputs of each pair of guidance signals leads to improvements (in terms of ROUGE-1), indicating every pair of guidance complements each other. The underlined results are the model performance without combinations.
score of each output of the four guidance signals and pick the best one. As shown in Table 6, despite the fact that the highlighted sentence signal achieves the best overall performance, it still underperforms one of the other three varieties of guidance more than  $60\%$  of the time. In addition, by aggregating their best outputs together, we can achieve a ROUGE-1/L point of 48.30/45.15, which significantly outperforms any single guided model. Further, we try to aggregate these guidance signals in a pairwise manner, and Table 7 demonstrates that each guidance signal is complementary to each other to some extent. Thus, we can safely conclude that each type of guidance signal has its own merits and one promising direction is to utilize a system
combination method such as Hong et al. (2015) to aggregate the results together.
Controllability. It is also of interest what effect this guidance has on the model outputs qualitatively. We sample several generated outputs (Table 8) and find that different provided signals can result in different outputs. Especially, for our sentence-guided model, providing the model with by running tissue paper over his son seth makes him sleep enables the model to generate the exact same sentence, and when the model is fed with one grateful viewer of the video commented..., it will generate one viewer commented.... The examples demonstrate that our model can generate summaries mostly faithful to the guidance signals while also performing abstraction.
Faithfulness of Generated Summaries. We also evaluate whether our generated summaries are faithful to the source document. We randomly sample 100 datapoints from the test set and ask 3 people from Amazon Mechanical Turk to evaluate their factual correctness. Each person gives a score between 1 and 3, with 3 being perfectly faithful to the source document. Table 9 shows that our guided model can generate more faithful summaries compared with the baseline.
Necessity of Using Oracles During Training. As mentioned previously, we use an oracle to select guidance signals during training. In this part, we investigate if we can provide automatically constructed guidance to the model during training as well. Table 10 shows that this methodology will lead to significantly worse performance. We con
Table 8: Different guidance signals lead to qualitatively distinct outputs. Tokens that appear in both the guidance and output are bolded. Sentences generated differently because of the effect of guidance content are italicized.
BertAbs	Ours
Sentence	Keyword	Relation	Retrieve
2.117	2.393*	2.347*	2.303*	2.310*
jecture that this is because when the relevancy between guidance and reference is weakened, the model will not learn to depend on the guidance signals and thus the model will be reduced to the original abstractive summarization baseline.
We propose a general framework for guided neural summarization, using which we investigate four types of guidance signals and achieve state-of-the-art performance on various popular datasets. We demonstrate the complementarity of the four guid
Table 9: Human evaluation of the faithfulness of different model outputs. * indicates significant improvements  $(p < 0.001)$  over baseline with using bootstrap.
Train	Test	R-1	R-2	R-L
Oracle	Auto	43.78	20.66	40.66
Oracle	55.18	32.54	52.06
Auto	Auto	41.61	19.04	38.65
Oracle	43.07	20.79	40.13
Table 10: Using automatically constructed guidance during training degrades the performance significantly.
ance signals, and find that our models can generate more novel words and more faithful summaries. We also show that we can control the output by providing user-specified guidance signals.
Given the generality of our framework, this opens the possibility for several future research directions including 1) developing strategies to ensemble models under different guidance signals; 2) incorporating sophisticated techniques such as copy or coverage over the source document, the guidance signal, or both; and 3) experimenting with other kinds of guidance signals such as salient
elementary discourse units.
We thank Shruti Rijhwani, Yiran Chen, Jiacheng Xu and anonymous reviewers for valuable feedback and helpful suggestions. This work was supported in part by a grant under the Northrop Grumman SO-TERIA project and the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.
Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018. Retrieve, rerank and rewrite: Soft template based neural summarization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the Workshop on Neural Machine Translation and Generation (WNGT).
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Pierre-Etienne Genest and Guy Lapalme. 2012. Fully abstractive approach to guided summarization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
Kai Hong, Mitchell Marcus, and Ani Nenkova. 2015. System combination for multi-document summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020. Semsum: Semantic dependency guided neural abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of reddit posts with multi-level memory networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Mahnaz Koupaee and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. arXiv preprint.
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Chenliang Li, Weiran Xu, Si Li, and Sheng Gao. 2018. Guiding generation for abstractive text summarization based on key information guide network. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018a. Generating wikipedia by summarizing long sequences. In Proceedings of the Intertional Conference on Learning Representatinos (ICLR).
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the Coference on Empirical Methods in Natural Language Processing (EMNLP).
Yizhu Liu, Zhiyi Luo, and Kenny Zhu. 2018b. Controlling length in abstractive summarization using a convolutional neural network. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of the SIGNLL Conference on Computational Natural Language Learning (CoNLL).
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018a. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Karolina Owczarzak and Hoa Trang Dang. 2010. Overview of the TAC 2010 summarization track. In Proceedings of the Text Analysis Conference (TAC).
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the Intertional Conference on Learning Representatins (ICLR).
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, and Junji Tomita. 2020. Abstractive summarization with combination of pre-trained sequence-to-sequence and saliency models. arXiv preprint.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2020. Boosting factual correctness of abstractive summarization with knowledge graph. arXiv preprint.
A Greedy Selection Algorithm
Algorithm 1 demonstrates how we use an oracle to select a subset of source sentences that have
the highest ROUGE scores with the reference summary. We use a similar algorithm to select the relation triples as well. Concretely, we flatten each relational triple  $(s,r,o)$  by concatenating its elements together and treat each concatenated text as a source sentence, then use Algorithm 1 to select the relation triples greedily.
We perform more analysis on CNN/DM in this section. Unless otherwise stated, we use oracle extractions at training time and BertAbs as our base model.
B.1 Controllability
In addition to the qualitative results in the main paper, we also perform a quantitative analysis to demonstrate the controllability of our models.
The quantitative results in Table 3 of the main text already demonstrate to some extent that we can control the model with guidance signals, as guidance signals of better quality can lead to better summaries. To further demonstrate this, we randomly sample guidance signals multiple times and plot the correlation between guidance quality and output quality in Figure 4. We can clearly see that there is a strong correlation between these two variables, indicating the controllability of our model.
Figure 4: There is a strong correlation between the guidance quality and output quality, demonstrating the controllability of our guided model.
Model	Ref.	Guidance	R-1	R-2	R-L
Sentence	1st	1st	49.49	29.39	46.25
2nd	28.66	10.09	26.05
2nd	1st	20.63	5.29	18.25
2nd	40.33	23.16	37.36
Keyword	1st	1st	40.52	21.06	37.54
2nd	33.35	14.67	30.60
2nd	1st	22.49	7.26	20.17
2nd	28.75	12.65	26.19
Relation	1st	1st	40.45	21.05	37.52
2nd	33.56	14.65	30.79
2nd	1st	22.85	7.47	20.47
2nd	28.48	12.42	25.89
Retrieve	1st	1st	39.32	19.74	36.32
2nd	33.89	15.29	31.14
2nd	1st	22.61	7.55	20.34
2nd	28.31	12.33	25.72
Table 11: We divide each summary reference into two halves and deduce the oracle guidance from them separately. Feeding incompatible guidance signals can lead to degraded performance.
In addition, we try to divide each test reference summary into two halves, then use oracle extraction to obtain guidance signals for both of these two halves and feed them to the model. Table 11 shows that feeding incompatible guidance signals can lead to degraded performance, which further demonstrates that we can control the summary through provision of user-specified inputs.
B.2 Semantic Similarity
To evaluate the semantic similarities between our model outputs and the reference, we also compute the METEOR scores (Banerjee and Lavie, 2005). As shown in Table 12, all of our guided models can outperform BertAbs in temrs of both of METEOR. However, it is surprising that BertExt achieves the
Model	METEOR	#Words (k)
exact match	+ stem/syn/para
BertExt	22.24	20.69	828.62
BertAbs	19.43	18.01	669.16
Ours
Sentence	20.21	18.88	626.73
Keyword	20.16	18.70	700.48
Relation	20.12	18.60	749.30
Retrieve	19.59	18.07	717.22
Table 12: Semantic similarity evaluation. We report results both in exact match mode (rewarding exact matches between words) and full mode (rewarding matching stems, synonyms and paraphrases as well).
Model	Train	Test	R-1	R-2	R-L
Sentence	Oracle	Auto	43.78	20.66	40.66
Oracle	55.18	32.54	52.06
Auto	Auto	41.61	19.04	38.65
Oracle	43.07	20.79	40.13
Keyword	Oracle	Auto	42.21	19.36	39.23
Oracle	45.08	22.22	42.07
Auto	Auto	41.72	19.15	38.78
Oracle	41.76	19.25	38.83
Relation	Oracle	Auto	41.40	18.66	38.40
Oracle	45.96	23.09	42.92
Auto	Auto	40.29	18.30	37.33
Oracle	40.67	18.41	37.70
Retrieve	Oracle	Auto	40.88	18.24	37.99
Oracle	43.69	20.53	40.71
Auto	Auto	40.86	18.5	37.95
Oracle	41.45	18.86	38.46
Table 13: Using automatically constructed guidance during training degrades the performance significantly.
best performance, possibly because METEOR has a tendency to favor long summaries.
B.3 Automatic Factual Correctness Evaluation
Besides human evaluation, we have also tried to use factCC (Krysciński et al., 2019)<sup>4</sup> to evaluate the factual correctness of our model outputs automatically. However, as shown in Figure 5, the factCC tool will give the gold reference an accuracy of about  $10\%$ . Considering our model is optimized towards the gold reference, the factCC score might not be a good indicator of whether there are factual errors in a generated summary.
Figure 5: The factCC model will give the gold reference an accuracy of about  $10\%$ .
B.4 Necessity of Using Oracles During Training
We have demonstrated in the main paper that it is necessary to use an oracle to select guidance signals during training for highlighted sentence models. In this part, we investigate if this is true for all the three guidance signals as well. Table 13 shows that this methodology will lead to significantly worse performance for other guidance signals as well, which further verifies our hypothesis that when the relevancy between guidance and reference is weakened, the model will not learn to depend on the guidance signals and thus the model will be reduced to the original abstractive summarization baseline.
B.5 Domain Adaptation.
We also evaluate the performance of our highlighted sentence-guided models under domain adaptation settings, namely train a summarization model on one dataset and test it on some other datasets. As shown in Table 14, generally, extractive models can outperform abstractive ones under domain adaptations settings and our model can achieve better performance than abstractive baselines. However, while our model is given the extracted sentences by the extractive model, we still cannot outperform extractive baselines. These negative results indicate that our model may still fail to fully depend on guidance signals when doing adaptation. Possible future directions include dropping out the input documents occasionally during training so that the model can learn to better condition on the guidance.
<sup>4</sup>https://github.com/salesforce/factCC
Method	CNNDM	XSUM	NYT
XSUM	NYT	CNNDM	NYT	CNNDM	XSUM
BertExt	20.55	2.84	15.55	44.80	24.35	41.37	35.98	13.38	32.56	37.35	16.67	33.84	40.18	17.21	36.40	19.93	2.75	14.94
BertAbs	20.39	2.85	15.89	40.99	20.41	37.91	26.31	5.54	21.80	20.60	3.75	16.53	35.77	14.24	32.67	16.11	2.24	12.85
Ours	20.55	2.89	16.00	43.55	21.83	40.51	26.72	5.62	22.08	23.74	3.61	18.37	36.23	14.37	33.15	16.14	2.14	12.92
Table 14: Performance of sentence-guided model under domain adaptation settings. The first row and second row represent source and target domains respectively.