SlowGuess/ACL_2020_OCR huggingface dataset:
We tune hyperparameters on the development data of the headline corpus, and set the weighting parameter  $\gamma$  to 12 for all models. The search steps and restarts are set to  $\beta_{T} = 0.1$  and  $\beta_{R} = 0.035$ , respectively. We see a sharp performance improvement when we do more searching. Thus, we choose  $\beta_{T}$  and  $\beta_{R}$  at the critical values due to efficiency concerns.
5.2 Competing Models
Besides the Lead baselines discussed in Section 4.2, we compare our models with state-of-the-art unsupervised sentence summarization systems.
Wang and Lee (2018) $^{6}$  use cycle-consistency to reconstruct source sentences from the headline generation corpus (Rush et al., 2015). The latent discrete representation, learned to be similar to (non-parallel) headlines, is used as the summary.
Zhou and Rush (2019) optimize an objective function involving language fluency and contextual matching. Their language modeling scorer is trained on headlines of the Gigaword training set; their contextual matching scorer is based on ELMo embeddings (Peters et al., 2018) trained with the Billion Word corpus (Chelba et al., 2013). Their summary length is controlled by a soft length penalty during beam search.
Fevry and Phang (2018)<sup>7</sup> learn a denoising autoencoder (Vincent et al., 2008) to reconstruct source sentences of the Gigaword training set. Summary length is set to  $50\%$  of the input length and is controlled by length embeddings in the decoder.
Baziotis et al. (2019)<sup>8</sup> propose  $\mathsf{SEQ}^3$  that uses cycle-consistency to reconstruct source sentences from the Gigaword training set. The length is also set to  $50\%$  of the input length, controlled by length embeddings in the intermediate decoder.
For the DUC2004 dataset, TOPIARY (Zajic et al., 2004) is the winning system in the competition. They shorten the sentence by rule-based syntax-tree trimming (Dorr et al., 2003), but enhance the resulting summary with topics that are learned on
<sup>6</sup>Generated summaries are obtained via E-Mail correspondence. Scores differ because of evaluation setup.
7Retrained with official code (https://github.com/zhang/usc_dae) because the authors use a private test set.
Retrained with official code (https://github.com/cbaziotis/seq3), because of different test data. The authors remove 54 noisy instances. Our replication thus achieves slightly lower scores than theirs.
Model	Data	Len D	ROUGE F1	Len O
article	title	external	R-1	R-2	R-L
A	Lead-N-8	✓	8	21.39	7.42	20.03	7.9
HC_article_8	✓	8	23.09	7.50	21.29	7.9
HC_title_8	✓	8	26.32	9.63	24.19	7.9
B	Lead-N-10	✓	10	23.03	7.95	21.29	9.8
Wang and Lee (2018)	✓	✓	-	27.29	10.01	24.59	10.8
Zhou and Rush (2019)	✓	billion	-	26.48	10.05	24.41	9.3
HC_article_10	✓	10	24.44	8.01	22.21	9.8
HC_title_10	✓	10	27.52	10.27	24.91	9.8
HC_title+twitter_10	✓	twitter	10	28.26	10.42	25.43	9.8
HC_title+billion_10	✓	billion	10	28.80	10.66	25.82	9.8
C	Lead-P-50	✓	50%	24.97	8.65	22.43	14.6
Fevry and Phang (2018)	✓	SNLI	50%	23.16	5.93	20.11	14.8
Baziotis et al. (2019)	✓	50%	24.70	7.97	22.14	15.1
HC_article_50p	✓	50%	25.58	8.44	22.66	14.9
HC_title_50p	✓	50%	27.05	9.75	23.89	14.9
Table 1: Results for headline generation on the Gigaword test set. Data: data used during training (source article, target titles, external corpus). billion: the Billion Word Corpus (Chelba et al., 2013); twitter: the Twitter corpus (Pagliardini et al., 2018); SNLI: the Stanford Natural Language Inference dataset (Bowman et al., 2015). Len D: desired summary length. ROUGE F1 (R-1, R-2, R-L): ROUGE-1, ROUGE-2, ROUGE-L F1 scores. Len O: averaged output length. Best results in bold. Second best results underlined. A: Models with output length around 8 words. B: Models with output length around 10 words. C: Models with output length around  $50\%$  of the input. Our hill-climbing (HC) approaches are named in the format of HC_data_outputLength.
Model	ROUGE Recall
R-1	R-2	R-L
Lead-C-75	22.50	6.49	19.72
SEQ3 (Baziotis et al., 2019)	22.13	6.18	19.3
TOPIARY (Zajic et al., 2004)	25.12	6.46	20.12
BOTTLESUM Ex (West et al., 2019)	22.85	5.71	19.87
HC_article_13	24.21	6.63	21.24
HC_title_13	26.04	8.06	22.90
HC_title+twitter_13	27.41	8.76	23.89
Table 2: Results on the DUC2004 dataset.
full articles.
BOTTLESUM EX (West et al., 2019) uses the information bottleneck principle to predict the next sentence in an article. Their method employs a pretrained small GPT-2 model (Radford et al., 2019).
Results for Headline Generation. We first compare with Lead-N-8 (Group A, Table 1). This is a standard baseline in previous work, because the average reference summary contains eight words. Unfortunately, none of the previous papers consider output length during evaluation, making comparisons between their (longer) output summaries and the Lead-N-8 baseline unfair, as discussed in Section 4.4. Our approach, which explicitly controls summary length, considerably outperforms the Lead-N-8 baseline in a fair setting.
Next, we compare with state-of-the-art unsupervised methods, whose output summary has roughly 10 words on average (Group B). In this case, we
set our hard length constraint as 10 and include the Lead-N-10 baseline for comparison. Trained on the title side only, our HC_title_10 model outperforms these competing methods in all ROUGE F1 scores. In particular, Zhou and Rush (2019) use the target side to train the language model, plus the Billion Word Corpus to pretrain embeddings used in the contextual matching scorer. With the same extra corpus to pretrain our sent2vec embeddings, our HC_title+billion_10 variant achieves even better performance, outperforming Zhou and Rush (2019) by 2.32 ROUGE-1 and 1.41 ROUGE-L points.
The Billion Word Corpus, however, includes complete articles, which implicitly yields unaligned parallel data. This could be inappropriate for an unsupervised method. Thus, we further train sent2vec embeddings on the Twitter corpus by Pagliardini et al. (2018). The HC_title+twitter_10 also performs better than HC_title_10 and other competitors.
In Group C, we compare with the models whose summaries have an average length of  $50\%$  of the input sentence. We set our desired target length to  $50\%$  as well, and include the Lead-P-50 baseline. Previous studies report a performance improvement over the Lead-N-8 baseline, but in fact, Table 1 shows that they do not outperform the appropriate Lead baseline Lead-P-50. Our model is the only unsupervised summarization system that outperforms the Lead-P-50 baseline on this dataset,
even though it is trained solely on the article side.
It is noted that our models trained on the title side (HC_title) consistently outperform those trained on the article side (HC_article). This is not surprising because the former can generate headlines from the learned target distribution. This shows the importance of learning a summary language model even if we do not have supervision of parallel source-target data.
Results for DUC2004. Table 2 shows the results on the DUC2004 data. As this dataset is for test only, we directly transfer the models HC_article and HC_title from the headline generation corpus with the same hyperparameters (except for length). As shown in the table, we outperform all previous methods and the Lead-C-75 baseline. The results are consistent with Table 1, showing the generalizability of our approach.
Human Evaluation. We conduct human evaluation via pairwise comparison of system outputs, in the same vein as (West et al., 2019). The annotator sees the source sentence along with the headline generated by our system and a competing method, presented in random order. The annotator is asked to compare the fidelity and fluency of the two systems, choosing among the three options (i) the first headline is better (ii) the second headline is better, and (iii) both headlines are equally good/bad. This task is repeated for 100 instances with 5 annotators each. The final label is selected by majority voting. The inter-annotator agreement (Krippendorff's alpha) is 0.25 when our model is compared with Wang and Lee (2018) and 0.17 with Zhou and Rush (2019).
We report the aggregated score of our system in Table 3. For each sample, we count 1 point if our model wins, 0 points if it ties, -1 point if it loses. The points are normalized by the number of samples. The results show an advantage of our model over Wang and Lee (2018), especially in fluency. Our model is also on par with Zhou and Rush (2019). Note again that we achieve this with fewer data.
In this section, we conduct an in-depth analysis of our model, based on HC_title_10 for headline generation.
Search Objective. Table 4 provides an ablation study on our objective function. It shows that both language fluency and semantic similarity play a
Models	Score (#wins/#ties/#loses)
Fidelity	Fluency
HC vs. WL	+0.18 (44/30/26)	+0.30 (45/40/15)
HC vs. ZR	+0.05 (35/35/30)	-0.03 (24/49/27)
Table 3: Human evaluation in a pairwise comparison setting on 100 headline generation instances. We show the scores of our model (HC_title_10) when it is compared with WL (Wang and Lee, 2018) and ZR (Zhou and Rush, 2019), in terms of average score of fidelity and fluency: 1 (wins), 0 (ties), and -1 (losses).
Objective	ROUGE F1 scores
f=	R-1	R-2	R-L
fL→M·fSIM(full model)	27.52	10.27	24.91
fL→M·fSIM	27.50	10.15	24.79
fL→M	25.24	8.87	23.09
fL→M	25.18	8.72	22.93
fSIM	20.31	4.08	18.19
Table 4: Ablation study of the search objective. Model HC_title_10 on the headline generation test set. Length constraint term omitted from notation.
role in measuring the quality of a summary. The bi-directional language model is also slightly better than a uni-directional language model.
Search Algorithm. In Figure 3, we compare our FCHC with the theoretical optimum on short sentences where exhaustive search is tractable. For only  $3\%$  of the instances with source sentence length between 25 and 30 words, our FCHC algorithm does not find the global optimum. In  $21\%$  of those cases, the better objective score leads to a higher ROUGE-L score. This shows that FCHC with restarts is a powerful enough search algorithm for word extraction-based sentence summarization.
Positional Bias. We analyze the positional bias of each algorithm by plotting the normalized frequency of extracted words within four different areas of the source sentence. As shown in Figure 4, the extraction positions of words in the reference headlines are slightly skewed towards the beginning of the source sentence. Our hill-climbing algorithm performs distributed edits over the sentence, which is reflected in the flatter graph across the source sentence areas. By contrast, beam search (Zhou and Rush, 2019) is more biased towards the first quarter of the source sentence. Cycle consistency models (Wang and Lee, 2018; Baziotis et al., 2019) show a strong bias towards the first half of the source sentence. We suspect that the reconstruction decoder is easily satisfied with the beginning of the source sentence as the discrete latent variable,
Figure 3: Orange crosses show the objective score optimized by exhaustive search minus the objective score optimized by FCHC. Blue pluses show the ROUGE-L difference between exhaustive search and FCHC. Plotted for the 1135 instances in the headline generation test set, where the source sentence has 30 words or fewer.
Figure 4: Positional bias for different systems, calculated for the headline generation test set. The source sentence is divided into four areas:  $0 - 25\%$ ,  $25 - 50\%$ ,  $50 - 75\%$ , and  $75 - 100\%$  of the sentence. The  $y$ -axis shows the normalized frequency of how often a word in the summary is extracted from one of the four source sentence areas.
because of its autoregressive decoding.
Case Study. We show example summaries generated by our system in Figure 5. We see that the HC_title models indeed learn the style of headlines, known as headline. As shown, HC_title often uses simple tense and drops articles (e.g., "a" and "the"). The summaries generated by HC_article tend to waste word slots by including an uninformative determiner.
It is also seen that we can control the length in an explicit way. Comparing HC_title with desired lengths of 8 and 10, we see that the additional two words are used to include more information, such as the day of the meeting in Example 2 or the gender of the injured person in Example 3.
1. Input: a german registered container ship ran aground at the entrance to the french port of le havre early tuesday, but authorities said there were no casualties.
Reference: container ship runs aground in french port
HC_article_10: a container ship ran aground but there were no casualties
HC title_10: container ship ran aground at french port but no casualties
HC title_8: ship ran aground at french port no casualties
2. Input: fidel castro, cuba's president of the council of state, met with a chinese delegation here tuesday.
Reference: castro meets chinese official
HC_article_10: fidel castro cuba's president met with a chinese delegation
HC_title_10: fidel castro cuba's president met with chinese delegation tuesday
HC_title_8: fidel castro's president met with chinese delegation
3. Input: two grenades exploded near a national police station Monday, slightly injuring one woman, news reports said.
Reference: two grenades explode near spanish police station HC_article_10: two grenades exploded near a police station injuring one woman
HC title 10: two grenades exploded near a police station injuring one woman
HC_title_8: two grenades exploded near police station injuring one
Table 5: Example summaries for headline generation test set.
We proposed a novel word-extraction model for sentence summarization that generates summaries by optimizing an objective function of language fluency and semantic similarity. A hard length constraint is also imposed in our objective function. In a controlled experiment, our model achieves better performance than strong baselines on headline generation and DUC2004 datasets.
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), under grant Nos. RGPIN-2019-04897, and RGPIN-2020-04465. Lili Mou is also supported by AltaML, the Amii Fellow Program, and the Canadian CIFAR AI Chair Program. This research was enabled in part by the support of Compute Canada (www.computecanada.ca).
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In EMNLP, pages 936-945.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben
gio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. 2019.  $\mathrm{SEQ}^3$ : Differentiable sequence-to-sequence-to-sequence autoencoder for unsupervised abstractive sentence compression. In *NAACL-HLT*, pages 673-681.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP, pages 632-642.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Philipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005.
Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In NAACL-HLT, pages 93-98.
James Clarke and Mirella Lapata. 2006. Constraint-based sentence compression: an integer programming approach. In COLING-ACL, volume 2, pages 144-151.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, pages 4171–4186.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-Interpreter model for sentence simplification through explicit editing. In ACL, pages 3393-3402.
Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In HLT-NAACL Workshop on Text Summarization, pages 1-8.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. JAIR, 22(1):457-479.
Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In ACL, pages 1074-1084.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proc. 2nd Workshop on Neural Machine Translation and Generation, pages 45-54.
Thibault Fevry and Jason Phang. 2018. Unsupervised sentence compression using denoising autoencoders. In CoNLL, pages 413-422.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In EMNLP, pages 4098-4109.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In ACL, pages 1631-1640.
Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In HLT-NAACL, pages 362-370.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In NIPS, pages 820-828.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. In ACL, pages 1535-1546.
Chiori Hori and Sadaoki Furui. 2004. Speech summarization: An approach through word extraction and a method for evaluation. IEICE Trans. Inf. & Syst., 87(1):15-25.
Tom Kenter, Alexey Borisov, and Maarten de Rijke. 2016. Siamese CBOW: Optimizing word embeddings for sentence representations. In ACL, pages 941-951.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In EMNLP, pages 1328-1338.
Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vechtomova. 2020. Iterative edit-based unsupervised sentence simplification. In ACL.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In ACL Workshop: Text Summarization Branches Out, pages 74-81.
Xianggen Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song. 2020. Unsupervised paraphrasing by simulated annealing. In ACL.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. CGMH: Constrained sentence generation by Metropolis-Hastings sampling. In AAAI, pages 6834-6842.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In EMNLP, pages 404-411.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI, pages 3075-3081.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In CoNLL, pages 280-290.
P Over and J Yen. 2004. An introduction to DUC-2004: Intrinsic evaluation of generic news text summarization systems. In Proc. Document Understanding Conference.
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embeddings using compositional n-gram features. In NAACL-HLT, pages 528-540.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In ICLR.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In *NAACL-HLT*, pages 2227-2237.
Dragomir R. Radev, Hongyan Jing, and Malgorzata Budzikowska. 2000. Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In NAACL-ANLP 2000 Workshop: Automatic Summarization.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog.
Colin Raffel, Minh-Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. In ICML, pages 2837-2846.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In EMNLP, pages 379-389.
Stuart J Russell and Peter Norvig. 2016. Artificial Intelligence: A Modern Approach. Pearson Education Limited.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL, pages 1073-1083.
Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. How to compare summarizers without target length? Pitfalls, solutions and re-examination of the neural summarization literature. In Proc. Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 21-29.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In ICML, pages 1096-1103.
Kai Wang, Xiaojun Quan, and Rui Wang. 2019. BiSET: Bi-directional selective encoding with template for abstractive summarization. In ACL, pages 2153-2162.
Yaushian Wang and Hung-yi Lee. 2018. Learning to encode text as human-readable summaries using generative adversarial networks. In EMNLP, pages 4187-4195.
Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. 2019. BottleSum: Unsupervised and self-supervised sentence summarization using the information bottleneck principle. In EMNLP-IJCNLP, pages 3750-3759.
Lei Yu, Jan Buys, and Phil Blunsom. 2016. Online segment to segment neural transduction. In EMNLP, pages 1307-1316.
David Zajic, Bonnie Dorr, and Richard Schwartz. 2004. BBN/UMD at DUC-2004: Topiary. In HLT-NAACL Document Understanding Workshop, pages 112-119.
Hao Zheng and Mirella Lapata. 2019. Sentence centrality revisited for unsupervised summarization. In ACL, pages 6236-6247.
Jiawei Zhou and Alexander M Rush. 2019. Simple unsupervised summarization by contextual matching. In ACL, pages 5101-5106.
Liang Zhou and Eduard Hovy. 2004. Template-filtered headline summarization. In ACL Workshop: Text Summarization Branches Out, pages 56-60.