SlowGuess/ACL_2020_OCR huggingface dataset:
Language Fluency We fine-tuned the GPT-2 medium model (Radford et al., 2019) on our collected headlines and then used it to measure the perplexity (PPL) on the generated outputs.
We used the fairseq code base (Ott et al., 2019). During training, we use Adam optimizer with an initial learning rate of  $5 \times 10^{-4}$ , and the batch size is set as 3072 tokens for each GPU with the parameters update frequency set as 4. For the random corruption for DAE training, we follow the standard practice to randomly delete or blank the word with a uniform probability of 0.2, and randomly shuffled the word order within 5 tokens. All datasets are lower-cased.  $\lambda$  is set as 0.5 in experiments. For each iteration of training, we randomly draw a batch of data either from the source dataset or from the target style corpus, and the sampling strategy follows the uniform distribution with the probability being equal to  $\lambda$ .
5.1 Human Evaluation Results
The human evaluation is to have a comprehensive measurement of the performances. We conduct experiments on four criteria, relevance, attraction, fluency, and style strength. We summarize the human evaluation results on the first three criteria in Table 2, and the last criteria in Table 4. Note that through automatic evaluation, the baselines NST, Fine-tuned, and Gigaword-MASS perform poorer than other methods (in Section 5.2), thereby we removed them in human evaluation to save unnecessary work for human raters.
Relevance We first look at the relevance scores in Table 2. It is interesting but not surprising that the pure summarization model NHG achieves the highest relevance score. The outputs from NHG
<sup>9</sup>https://github.com/pltrdy/files2rouge
10https://github.com/Maluuba/nlg-eval
PPL on the development set is 42.5
Style	Settings	Relevance	Attraction	Fluency
None	NHG	6.21	8.47	9.31
Human	5.89	8.93	9.33
Humor	Multitask	5.51	8.61	9.11
TitleStylist	5.87	8.93	9.29
Romance	Multitask	5.67	8.54	8.91
TitleStylist	5.86	8.87	9.14
Clickbait	Multitask	5.67	8.71	9.21
TitleStylist	5.83	9.29	9.44
Table 2: Human evaluation on three aspects: relevance, attraction, and fluency. "None" represents the original headlines in the dataset.
are usually like an organic reorganization of several keywords in the source context (as shown in Table 3), thus appearing most relevant. It is noteworthy that the generated headlines of our TitleStylist for all three styles are close to the original human-written headlines in terms of relevance, validating that our generation results are qualified in this aspect. Another finding is that more attractive or more stylistic headlines would lose some relevance since they need to use more words outside the news body for improved creativity.
Attraction In terms of attraction scores in Table 2, we have three findings: (1) The human-written headlines are more attractive than those from NHG, which agrees with our observation in Section 1. (2) Our TitleStylist can generate more attractive headlines over the NHG and Multitask baselines for all three styles, demonstrating that adapting the model to these styles could improve the attraction and specialization of some parameters in the model for different styles can further enhance the attraction. (3) Adapting the model to the "Clickbait" style could create the most attractive headlines, even out-weighting the original ones, which agrees with the fact that click-baity headlines are better at drawing readers' attention. To be noted, although we learned the "Clickbait" style into our summarization system, we still made sure that we are generating relevant headlines instead of too exaggerated ones, which can be verified by our relevance scores.
Fluency The human-annotated fluency scores in Table 2 verified that our TitleStylist generated headlines are comparable or superior to the human-written headlines in terms of readability.
Style Strength We also validated that our TitleStylist can carry more styles compared with the
Multitask and NHG baselines by summarizing the percentage of choices by humans for the most humorous or romantic headlines in Table 4.
5.2 Automatic Evaluation Results
Apart from the human evaluation of the overall generation quality on four criteria, we also conducted a conventional automatic assessment to gauge only the summarization quality. This evaluation does not take other measures such as the style strength into consideration, but it serves as important complimentary proof to ensure that the model has an acceptable level of summarization ability.
Table 5 summarizes the automatic evaluation results of our proposed TitleStylist model and all baselines. We use the summarization-related evaluation metrics, i.e., BLEU, ROUGE, CIDEr, and METEOR, to measure how relevant the generated headlines are to the news articles, to some extent, by comparing them to the original human-written headlines. In Table 5, the first row "NHG" shows the performance of the current state-of-the-art summarization model on our data, and Table 3 provides two examples of its generation output. Our ultimate goal is to generate more attractive headlines than these while maintaining relevance to the news body.
From Table 5, the baseline Gigaword-MASS scored worse than NHG, revealing that directly applying an off-the-shelf headline generation model to new in-domain data is not feasible, although this model has been trained on more than 20 times larger dataset. Both NST and Fine-tuned baselines present very poor summarization performance, and the reason could be that both of them cast the problem into two steps: summarization and style transfer, and the latter step is absent of the summarization task, which prevents the model from maintaining its summarization capability.
In contrast, the Multitask baseline involves the summarization and style transfer (via reconstruction training) processes at the same time and shows superior summarization performance even compared with NHG. This reveals that the unsupervised reconstruction task can indeed help improve the supervised summarization task. More importantly, we use two different types of corpora for the reconstruction task: one consists of headlines that are similar to the news data for the summarization task, and the other consists of text from novels that are entirely different from the news data. However,
News Abstract	Turkey's bitter history with Kurds is figuring prominently in its calculations over how to deal with Bush administration's request to use Turkey as the base for thousands of combat troops if there is a war with Iraq; Recep Tayyip Erdogan, leader of Turkey's governing party, says publicly for the first time that future of Iraq's Kurdish area, which abuts border region of Turkey also heavily populated by Kurds, is weighing heavily on negotiations; Hints at what Turkish officials have been saying privately for weeks: if war comes to Iraq, overriding Turkish objective would be less helping Americans topple Saddam Hussein, but rather preventing Kurds in Iraq from forming their own state.	Reunified Berlin is commemorating 40th anniversary of the start of construction of Berlin wall, almost 12 years since Germans jubilantly celebrated reopening between east and west and attacked hated structure with sledgehammers; Some Germans are championing the preservation of wall at the time when little remains beyond few crumbling remnants to remind Berliners of unhappy division that many have since worked hard to heal and put behind them; What little remains of physical wall embodies era that Germans have yet to resolve for themselves; They routinely talk of 'wall in the mind' to describe social and cultural differences that continue to divide easterners and westerners.
Human NHG	Turkey assesses question of Kurds	The wall Berlin can't quite demolish
Turkey's bitter history with Kurds	Construction of Berlin wall is commemorated
Humor
Romance
Clickbait	What if there is a war with Kurds?	The Berlin wall, 12 years later, is still there?
What if the Kurds say "No" to Iraq?	The Berlin wall: from the past to the present
For Turkey, a long, hard road	East vs West, Berlin wall lives on
Table 3: Examples of style-carrying headlines generated by TitleStylist.
Style	NHG	Multitask	TitleStylist
Humor	18.7	35.3	46.0
Romance	24.7	34.7	40.6
Clickbait	13.8	35.8	50.4
Table 4: Percentage of choices (%) for the most humorous or romantic headlines among TitleStylist and two baselines NHG and Multitask.
unsupervised reconstruction training on both types of data can contribute to the summarization task, which throws light on the potential future work in summarization by incorporating unsupervised learning as augmentation.
We find that in Table 5 TitleStylist-F achieves the best summarization performance. This implicates that, compared with the Multitask baseline where the two tasks share all parameters, specialization of layer normalization and encoder-attention parameters can make  $G_{S}$  focus more on summarization.
It is noteworthy that the summarization scores for TitleStylist are lower than TitleStylist-F but still comparable to NHG. This agrees with the fact that the  $G_{T}$  branch more focuses on bringing in stylistic linguistic patterns into the generated summaries, thus the outputs would deviate from the pure summarization to some degree. However, the relevance degree of them remains close to the baseline NHG, which is the starting point we want to improve on. Later in the next section, we will further validate that these headlines are faithful to the new article through human evaluation.
We also reported the perplexity (PPL) of the generated headlines to evaluate the language fluency, as shown in Table 5. All outputs from baselines NHG and Multitask and our proposed TitleStylist show similar PPL compared with the test set (used in the fine-tuning stage) PPL 42.5, indicating that they are all fluent expressions for news headlines.
5.3 Extension to Multi-Style
We progressively expand TitleStylist to include all three target styles (humor, romance, and clickbait) to demonstrate the flexibility of our model. That is, we simultaneously trained the summarization task on the headlines data and the DAE task on the three target style corpora. And we made the layer normalization and encoder-attention parameters specialized for these four styles (fact, humor, romance, and clickbait) and shared the other parameters. We compared this multi-style version, TitleStylist-Versatile, with the previously presented single-style counterpart, as shown in Table 6. From this table, we see that the BLEU and ROUGE-L scores of TitleStylist-Versatile are comparable to TitleStylist for all three styles. Besides, we conducted another human study to determine the better headline between the two models in terms of attraction, and we allow human annotators to choose both options if they deem them as equivalent. The result is presented in the last column of Table 6, which shows that the attraction of TitleStylist-Versatile outputs is competitive to TitleStylist. TitleStylist-Versatile thus generates multiple headlines in different styles altogether, which is a novel and efficient
Style Corpus	Model	BLEU	ROUGE-1	ROUGE-2	ROUGE-L	CIDEr	METEOR	PPL (↓)	Len. Ratio (%)
None	NHG	12.9	27.7	9.7	24.8	0.821	0.123	40.4	8.9
Gigaword-MASS	9.2	22.6	6.4	20.1	0.576	0.102	65.0	9.7
Humor	NST	5.8	17.8	4.3	16.1	0.412	0.078	361.3	9.2
Fine-tuned	4.3	15.7	3.4	13.2	0.140	0.093	398.8	3.9
Multitask	14.7	28.9	11.6	26.1	0.995	0.134	40.0	9.5
TitleStylist	13.3	28.1	10.3	25.4	0.918	0.127	46.2	10.6
TitleStylist-F	15.2	29.2	11.6	26.3	1.022	0.135	39.3	9.7
Romance	NST	2.9	9.8	0.9	9.0	0.110	0.047	434.1	6.2
Fine-tuned	5.1	18.7	4.5	16.1	0.023	0.128	132.2	2.8
Multitask	14.8	28.7	11.5	25.9	0.997	0.132	40.5	9.7
TitleStylist	12.0	27.2	10.1	24.4	0.832	0.134	40.1	7.4
TitleStylist-F	15.0	29.0	11.7	26.2	1.005	0.134	39.0	9.8
Clickbait	NST	2.5	8.4	0.6	7.8	0.089	0.041	455.4	6.3
Fine-tuned	4.7	17.3	4.0	15.0	0.019	0.116	172.0	2.8
Multitask	14.5	28.3	11.2	25.5	0.980	0.132	38.5	9.7
TitleStylist	11.5	26.6	9.8	23.7	0.799	0.134	40.7	7.3
TitleStylist-F	14.7	28.6	11.4	25.9	0.981	0.133	38.9	9.6
Table 5: Automatic evaluation results of our TitleStylist and baselines. The test set of each style is the same, but the training set is different depending on the target style as shown in the "Style Corpus" column. "None" means no style-specific dataset, and "Humor", "Romance" and "Clickbait" corresponds to the datasets we introduced in Section 4.1.2. During the inference phase, our TitleStylist can generate two outputs: one from  $G_{T}$  and the other from  $G_{S}$ . Outputs from  $G_{T}$  are style-carrying, so we denote it as "TitleStylist"; outputs from  $G_{S}$  are plain and factual, thus denoted as "TitleStylist-F." The last column "Len. Ratio" denotes the average ratio of abstract length to the generated headline length by the number of words.
Style	Model	BLEU	RG-L	Pref. (%)
None	TitleStylist-Versatile	14.5	25.8	—
Humor	TitleStylist-Versatile	12.3	24.5	42.6
TitleStylist	13.3	25.4	57.4
Romance	TitleStylist-Versatile	12.0	24.2	46.3
TitleStylist	12.0	24.4	53.7
Clickbait	TitleStylist-Versatile	13.1	24.9	52.9
TitleStylist	11.5	23.7	47.1
Table 6: Comparison between TitleStylist-Versatile and TitleStylist. "RG-L" denotes ROUGE-L, and "Pref." denotes preference.
We have proposed a new task of Stylistic Headline Generation (SHG) to emphasize explicit control of styles in headline generation for improved attraction. To this end, we presented a multitask framework to induce styles into summarization, and proposed the parameters sharing scheme to enhance both summarization and stylization capabilities. Through experiments, we validated our proposed TitleStylist can generate more attractive headlines than state-of-the-art HG models.
We appreciate all the volunteer native speakers (Shreya Karpoor, Lisa Orii, Abhishek Mohan, Paloma Quiroga, etc.) for the human evaluation of
our study, and thank the reviewers for their inspiring comments. Joey Tianyi Zhou is partially supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (Project No. A18A1b0045).
Michele Banko, Vibhu O Mittal, and Michael J Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 318-325. Association for Computational Linguistics.
Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018a. Retrieve, rerank and rewrite: Soft template based neural summarization. In ACL.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018b. Faithful to the original: Fact aware neural abstractive summarization. In Thirty-Second AAAI Conference on Artificial Intelligence.
Cheng-Kuan Chen, Zhu Feng Pan, Ming-Yu Liu, and Min Sun. 2019. Unsupervised stylish image description generation via domain layer norm. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 8151-8158. AAAI Press.
Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93-98.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197.
Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03 on Text summarization workshop-Volume 5, pages 1-8. Association for Computational Linguistics.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. 2016. A learned representation for artistic style. arXiv preprint arXiv:1610.07629.
HP Edmundson. 1964. Problems in automatic abstracting. Communications of the ACM, 7(4):259-263.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, NMT@ACL 2018, Melbourne, Australia, July 20, 2018, pages 45-54. Association for Computational Linguistics.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Yidi Guo, Heyan Huang, Yang Gao, and Chi Lu. 2017. Conceptual multi-layer neural network model for headline generation. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pages 355-367. Springer.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Tatsuru Higurashi, Hayato Kobayashi, Takeshi Masuyama, and Kazuma Murao. 2018. Extractive headline generation based on learning to rank for community question answering. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1742-1753.
Kango Iwama and Yoshinobu Kano. 2019. Multiple news headlines generation using page metadata. In Proceedings of the 12th International Conference on Natural Language Generation, 2019. Association for Computational Linguistics.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Unsupervised domain adaptation for neural machine translation with iterative back translation. arXiv preprint arXiv:2001.08140.
Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. Unsupervised text attribute transfer via iterative matching and translation. In IJCNLP 2019.
Hongyan Jing and Kathleen McKeown. 1999. The decomposition of human-written summary sentences.
Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018. Content selection in deep learning models of summarization. arXiv preprint arXiv:1810.12343.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3294-3302.
Guillaume Lample, Sandeep Subramanian, Eric Michael Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In ICLR.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Junyang Lin, Xu Sun, Shuming Ma, and Qi Su. 2018. Global encoding for abstractive summarization. In ACL.
Tianshang Liu, Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Review headline generation with user embedding. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Changsha, China, October 19-21, 2018, Proceedings, pages 324-334.
Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159-165.
Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multitask sequence to sequence learning. CoRR, abs/1511.06114.
Betty A Mathis, James E Rush, and Carol E Young. 1973. Improvement of automatic abstracts by the use of structural analysis. Journal of the American Society for Information Science, 24(2):101-109.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Dragomir R Radev and Kathleen R McKeown. 1998. Generating natural language summaries from multiple on-line sources. Computational Linguistics, 24(3):470-500.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685.
Alexander M Rush, SEAS Harvard, Sumit Chopra, and Jason Weston. 2017. A neural attention model for sentence summarization. In ACLWeb. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Gerard Salton, Amit Singhal, Mandar Mitra, and Chris Buckley. 1997. Automatic text structuring and summarization. Information processing & management, 33(2):193-207.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Shi-Qi Shen, Yan-Kai Lin, Cun-Chao Tu, Yu Zhao, Zhi-Yuan Liu, Mao-Song Sun, et al. 2017a. Recent advances on neural headline generation. Journal of computer science and technology, 32(4):768-784.
Shiqi Shen, Yun Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2018. Zero-shot cross-lingual neural headline generation. IEEE/ACM Trans. Audio, Speech & Language Processing, 26(12):2319-2327.
Shiqi Shen, Yu Zhao, Zhiyuan Liu, Maosong Sun, et al. 2016. Neural headline generation with sentence-wise optimization. arXiv preprint arXiv:1604.01904.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017b. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems